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EXECUTIVE  SUMMARY 

TOPICS  AND  GOAL  OF  THIS  REPORT 

This  Report  deals  with  current  issues  of  training: 

•  How  to  quicken  the  training  process  while  maintaining  its  effectiveness  (Rapidized 
Training), 

•  How  to  rapidize  the  transposition  of  lessons  learned  from  the  battlespace  into  the  training 
context  {Rapidized  Knowledge  Sharing), 

•  How  to  train  more  quickly  to  higher  levels  of  proficiency  {Accelerated  Proficiency),  and 

•  How  to  insure  that  training  has  a  stable  and  lasting  effect  {Facilitated  Retention). 


In  addition  to  a  literature  review,  this  Report  integrates  results  from  a  "Workshop  on  Accelerated 
Learning"  (June,  2008)  and  a  "Working  Meeting  on  Accelerated  Proficiency  and  Facilitated 
Retention"  (October,  2009),  both  sponsored  by  the  Human  Effectiveness  Directorate  and  the 
DoD  Accelerated  Learning  Technology  Focus  Team.  The  Workshops  brought  together  leading 
academic,  private  sector,  and  DoD  specialists  in  areas  of  training  and  expertise  studies. 

The  goal  of  accelerated  learning  calls  out  an  important  practical  tension.  On  the  one  hand  is  the 
notion  that  we  must  accelerate  learning;  that  is,  increase  the  rate  at  which  highly  proficient 
performance  is  achieved.  On  the  other  hand,  there  is  a  significant  amount  of  evidence  that 
developing  expertise  requires  up  to  10  years  of  experience,  including  practice  at  tough  tasks 
(Hoffman,  1998).  This  suggests  that  it  is  not  possible  to  accelerate  the  achievement  of  high 
proficiency.  A  prime  goal  of  the  DoD  Accelerated  Learning  Technology  Focus  Team  is  to 
identify  critical  research  challenges  that  are  currently  underfunded  or  not  funded,  and  generate  a 
notional  roadmap  for  science  and  technology  advancement. 

BACKGROUND 

Domain  practitioners  who  achieve  high  levels  of  proficiency  provide  technical  judgment  to  speed 
decision-making  in  time-critical  events.  They  provide  resilience  to  operations  by  resolving  tough 
problems,  acting  prudently  by  judgment  rather  than  by  rule,  and  anticipating  future  demands 
with  re-planning.  High  proficiency  individuals  exercise  effective  technical  leadership  in 
ambiguous  or  complex  situations,  often  by  communicating  subtle  features  that  other  people  will 
not  see  until  they  are  pointed  out.  It  typically  takes  years  of  experience  for  professionals  to 
master  their  domain.  Reasons  for  this  include  domain  complexity  and  the  need  for  extended  and 
continuing  practice  at  rare  and  difficult  cases. 

Across  recent  decades,  many  military  workplaces  have  changed  and  many  new  ones  emerged,  as 
forms  of  "complex  sociotechnical  systems"  in  which  the  work  is  cognitive  and  collaborative,  and 
heavily  reliant  on  computer  technology.  Work  in  such  domains  requires  high  levels  of 
proficiency,  in  terms  of  knowledge,  reasoning  skill,  and  critical  thinking  skill.  Specific  domains 
include  command  posts,  intelligence  analysis,  emergency  response,  disaster  relief,  and 
cyberdefense.  Within  the  military,  the  required  preparedness  status  includes  capabilities  to  be 
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adaptive,  resilient,  and  robust  in  the  face  of  unexpected  disruptions.  This  is  referred  to  as 
"cognitive  readiness"  (Morrison  &  Fletcher,  2002). 

Critical  military  competencies  that  ordinarily  take  years  to  develop  include  cultural 
understanding,  understanding  and  evaluation  of  political,  cultural  and  economic  environments  of 
the  battlespace,  military  implications  of  fused  sensing  and  intelligence,  space  management  for 
complex  operations  (i.e.,  avoiding  interference),  deception,  influence  and  information  operations, 
and  campaign  replanning  (Joint  Chiefs  of  Staff,  2005).  The  challenge  of  achieving  and 
maintaining  high  proficiency  is  compounded  in  the  military  by  such  practices  as  collateral 
assignment,  redeployment  (e.g.,  rapid  skill  decay  on  the  part  of  pilots  creates  a  need  for 
expensive  re-training),  inadequate  or  ad  hoc  mentoring,  and  the  drive  for  “just-in-time”  training. 

The  concept  of  accelerated  learning  was  proposed  in  1990  and  is  now  a  salient  notion  in  business 
management  and  education.  The  phenomenon  of  achieving  higher  levels  of  proficiency  in  less 
time  than  taken  in  traditional  training  has  been  demonstrated  in  business,  in  teacher  education, 
and  in  the  training  of  electronics  technicians.  The  U.S.  Army's  "Think  Like  a  Commander" 
program  and  the  National  Training  Center  have  demonstrated  accelerated  learning,  as  has  the 
U.S.  Navy's  "Top  Gun"  program.  A  number  of  new  DoD  programs  seek  to  achieve  accelerated 
learning,  such  as  the  Air  Force  Weather  Agency's  "Forecast  Ready"  program. 

Acceleration  in  the  sense  of  rapidized  training  has  also  been  demonstrated:  The  quickening  of 
training  without  reduction  in  the  proficiency  level  achieved  at  the  end  of  the  training.  Indeed, 
indications  are  that  rapidization  methods  can  lead  to  higher  levels  of  proficiency  than  legacy 
training. 

Acceleration  in  the  sense  of  achieving  higher  levels  of  proficiency  has  also  been  demonstrated, 
and  in  complex  military  jobs/tasks.  The  demonstrations  all  involve  capturing  the  knowledge  and 
reasoning  of  the  most  experienced  experts,  and  packaging  it  into  practice  problems  and 
scenarios.  As  a  side  benefit,  the  methodology  behind  such  demonstrations  also  illustrate 
rapidized  transposition  of  lessons  learned  into  the  training  context. 

LITERATURE  REVIEW 

One  must  be  cautious  in  generalizing  with  regard  to  the  effects  of  the  numerous  interacting 
variables  that  affect  training.  "Training  for  complex  tasks  is  itself  a  complex  task  and  most 
principles  for  good  instruction  are  contextual,  not  universal"  (Reigeluth,  personal 
communication).  Some  general  findings  are: 

Training 

•  Training  is  generally  more  effective  if  the  initial  learning  is  about  domain  concepts  and 
principles  rather  than  specific  details  of  tasks,  and  procedures. 

•  Training  at  advanced  levels  (training  to  perform  in  dynamic  and  complex  domains  where 
tasks  are  not  fixed)  is  generally  more  effective  if  it  involves  extensive  practice  on  realistic 
examples  or  scenarios  (e.g.,  problem-based  learning). 

•  Training  has  to  balance  giving  and  withholding  of  outcome  and  process  feedback  to 
achieve  optimal  learning  at  different  stages  of  advancement. 
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•  Training  at  intermediate  and  advaneed  levels  benefits  signifieantly  from  experienee  at 
challenging  problems  or  cases,  the  "desirable  difficulties."  Short-term  performance  might 
suffer,  but  longer-term  gains  will  emerge. 

•  Mentoring  is  valuable  and  critical  for  advanced  learning,  because  it  provides 
opportunities  to  receive  rich  process  and  outcome  feedback  as  learners  encounter 
increasingly  complex  problems.  However,  mentoring  is  not  always  necessary  in  advanced 
stages  of  learning. 

•  Training  using  intelligent  tutoring  systems  and  serious  games  (virtual  reality  systems)  can 
be  highly  effective. 

Transfer 

•  Initial  learning  that  is  more  difficult  can  lead  to  greater  flexibility  and  transfer;  when 
learners  are  initially  exposed  to  simplifications  of  complex  topics,  serious 
misunderstandings  can  remain  entrenched  and  interfere  with  or  preclude  deeper,  more 
accurate  understandings. 

•  Transferring  a  skill  to  new  situations  is  often  difficult  but  can  be  promoted  by  following  a 
number  of  training  principles:  Employing  deliberate  practice,  increasing  the  variability  of 
practice,  adding  sources  of  contextual  interference,  using  a  mixed  practice  schedule, 
distributing  practice  in  time,  and  providing  process  and  outcome  feedback  in  an  explicit 
analysis  of  errors. 

Retention  and  Decay 

•  There  is  a  beneficial  effect  of  spacing  on  learning  and  memory  when  the  goal  is  long 
retention  intervals. 

•  Retention  is  better  if  the  same  task  is  never  practiced  on  successive  trials,  but  is 
randomized  with  other  practices. 

•  Significant  decay  can  occur  even  within  relatively  short  time  frames  (days  to  weeks)  for 
any  form  of  skill  or  learned  material,  including  that  involved  in  military  tasks. 

•  The  best  predictor  of  skill  retention  following  a  hiatus  is  the  level  of  performance 
achieved  (including  over-learning)  prior  to  the  hiatus. 

•  The  variable  having  the  largest  impact  on  performance  after  a  retention  interval  is  the 
similarity  of  the  conditions  of  testing  to  those  of  the  training. 

•  Overlearning  is  the  prime  determinant  of  memory  and  skill  decay:  The  greater  the  degree 
of  overlearning  the  less  decay  and  the  less  rapid  the  decline. 


Teams 

•  Complex  cognitive  work  generally  is  conducted  by  collaborations  of  people  and 
technologies.  Individuals  who  are  more  proficient  at  problem  solving  place  a  high  value 
on  cooperation  and  engage  in  more  work-related  communication. 

•  A  team  of  experts  is  not  necessarily  an  expert  team.  In  high  functioning  team,  the  team 
members  develop  rich  mental  models  of  the  goals  and  activities  of  the  other  team 
members. 

•  Team  training  must  consider  content  knowledge,  perceptual-motor  skills,  reasoning  skills 
and  strategies,  coordination  and  collaboration  skills,  and  attitudes  appropriate  for 
teamwork. 
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•  Communication  is  key  to  process  and  performance.  High  performing  teams  effectively 
exchange  information  in  a  consensually  agreed-upon  manner  and  with  consistent  and 
meaningful  terminology,  and  are  careful  to  clarify  or  acknowledge  the  receipt  of 
information. 

None  of  the  above  generalizations  holds  uniformly;  for  all  of  the  above,  one  can  find  studies 
showing  opposite  effects,  or  no  effects.  For  example,  while  it  makes  sense  to  maintain 
consistency  of  team  membership,  the  mixing  of  teams  can  allow  team  members  to  gain  skill  at 
helping  to  forge  high-functioning  teams.  In  addition,  in  military  teams  membership  is  often 
necessarily  ad  hoc,  there  is  a  mixture  of  proficiency  levels,  and  team  membership  typically 
changes  frequently. 

The  extensive  literature  on  the  cognition  of  experts  provides  benchmarks  and  standards,  in  terms 
of  the  defining  features  of  individuals  who  achieve  high  levels  of  proficiency.  The  research 
literature  also  presents  proven  methods  for  scaling  proficiency  in  terms  of  performance 
measurement.  High  proficiency  in  the  military  is  defined  by  critical  skills  for  accomplishing  the 
mission  and  achieving  national  defense.  Achieving  that  requires  a  constant  'stretching'  of  the  skill 
(defined  by  increasing  challenges),  high  levels  of  intrinsic  motivation  to  work  hard  on  difficult 
problems,  practice  that  provides  rich,  meaningful  feedback,  and  practice  based  on  mentoring  or 
expert  instructional  guidance.  The  rule  of  thumb  is  that  it  takes  ten  years  of  experience  to  achieve 
expertise.  It  would  be  highly  desirable  to  shorten  that  span.  If  even  a  few  years  could  be  saved  in 
the  progression  from  apprentice  to  expert,  there  would  be  a  huge  savings  to  the  military. 

The  fact  that  learning,  training  and  instructional  design  are  so  highly  complex,  with  multiple 
interacting  factors,  makes  it  impossible  to  compose  a  recipe  for  instructional  design  for 
proficiency  training.  For  example,  Dodd  (2009)  discussed  the  implications  of  moving  target 
problems  for  military  training  and  "end-to-end"  evaluation.  In  an  ideal  situation,  one  would  be 
able  to  trace  the  effects  of  instruction,  from  the  very  beginning,  noting  its  effect  across  the  entire 
span  of  learning  various  knowledge  and  skills,  right  through  to  final  operations  performance. 
Reasons  why  this  is  not  possible  include: 

•  The  time  taken  for  the  changes  due  to  training  to  percolate  through  to  operational  effects 
can  be  long  and  involves  complex  interactions  with  other  functional  realities,  such  as 
organizational  constraints  and  operational  pressures. 

•  There  is  no  straightforward  track,  from  the  formal  process  of  lessons-learned  to  eventual 
operational  performance,  that  would  enable  us  to  give  credit  to  any  particular  training 
factor. 

•  Operational  outcomes  are  so  contextually  and  circumstantially  driven  that  they  defy 
backwards-looking  interpretation  necessary  to  understand  cause-and-effect  linkages. 

•  Operational  effectiveness  outcomes  are  tied  to  scenarios  and  circumstances,  making  it 
impossible  to  get  representative  coverage  permitting  generalizations  that  specify  which 
training  inputs  are  effective  in  meeting  which  possible  or  future  operational  challenges. 

WORKING  MEETING  RESUETS 

In  considering  the  meanings  of  accelerated  learning,  the  Working  Meeting  discussions  revealed  a 
host  of  paradoxes  that  work  against  notions  of  acceleration.  Eor  example,  as  technology  changes, 
as  assignments  change,  and  as  jobs  change,  the  cognitive  work  changes.  Eeaming  and  re-learning 
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on  the  job  must  be  continuous.  But  there  is  too  much  to  train,  in  too  little  time.  Thus,  the  pressure 
is  to  do  Just-in-Time  training,  but  such  training  is  at  odds  with  notions  of  proficiency  training. 
There  is  a  "paradox  of  tasks."  Traditionally,  training  depends  on  componential  analysis  of 
relatively  stable  tasks  that  can  be  described  as  series  of  steps  or  procedures.  But  cognitive  work 
is  the  context-sensitive,  knowledge-driven  choice  among  alternative  activities.  The  traditional 
concept  of  “task”  may  actually  be  an  impediment  to  advances  in  research  and  theory  in  that  it 
reinforces  an  artificial  notion  of  separability. 

The  most  salient  of  the  paradoxes  was  the  practice  of  rotating  the  most  highly-trained  and 
experienced  pilots  to  other  duties  and  responsibilities  and  then  sometimes  rotating  them  back 
into  piloting  or  pilot  training.  The  actual  achievement  of  acceleration  (in  any  of  its  senses)  will 
require  serious  consideration  of  certain  organization  traditions  and  practices.  Twelve  years  of 
operational  assignments  is  the  country’s  return  on  investment  for  all  the  training,  but  at  any  time 
after  two  years  at  the  principal  piloting  assignment,  an  individual  can  be  moved  off  that  career 
path  to  do  something  else,  typically  something  that  is  not  flying.  Reassigning  an  individual  just 
as  he  or  she  achieves  high  proficiency  cuts  against  the  notion  that  organizational  capability  is 
built  upon  individual  expertise.  Furthermore,  there  are  practical  barriers  to  refresher  training 
during  hiatus.  In  the  piloting  example,  although  there  is  the  possibility  of  periodic  practice  using 
a  simulator,  pilots  find  that  time  constraints  and  duty  requirements  often  preclude  opportunities 
to  practice. 

The  challenges  in  the  military  of  skill  decay  due  to  hiatus  assignments  are  not  unique  to  the 
military.  CEOs  and  senior  managers  can  get  cut  off  from  experiences  that  keep  them  fresh,  and 
then  they  make  bad  decisions.  Discussions  at  the  Working  Meeting  were  anchored  in  specific 
case  studies  of  diverse  professional  domains,  including  second  language  learning,  power 
generation  and  coordination,  weather  forecasting,  software  engineering,  STEM  fields, 
cyber  security,  cultural  awareness,  personnel  management,  and  general  command.  Special 
emphasis  was  placed  on  piloting  (fighters  and  UAVs),  in  the  Presentations  at  the  Working 
Meeting,  and  a  series  of  career  interviews  with  experienced  USAE  pilots.  Consensus  was  that  the 
military  has  need  for  more  individuals  who  can  perform  at  very  high  levels  of  proficiency  in  all 
of  the  areas  designated  by  COIN  (counterinsurgency),  DIME  (diplomatic,  informational,  military 
and  economic)  and  PIMESII  (political,  military,  economic,  social,  information  and 
infrastructure). 

A  number  of  innovative  concepts  were  introduced  at  the  Working  Meeting,  and  ideas  for  a 
program  of  research  were  proposed.  The  Working  Meeting  resulted  in  a  list  of  key 
considerations  in  the  design  of  a  research  and  technology  roadmap  for  establishing  methods  of 
accelerated  proficiency: 

•  Research  should  investigate  at  least  two  substantively  different  domains  that  have  civilian 
analogs,  relate  to  STEM  fields,  and  can  be  treated  as  "open  source"  for  getting  data  on 
baseline  performance. 

•  Research  should  investigate  domains  with  recently  transformed  strategic  environment, 
where  people  have  more  than  one  duty  assignment. 
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•  Research  must  establish  methods  for  rapidized  cognitive  task  analysis.  The  most 
frequently  mentioned  training  challenge  was  the  pace  of  change  in  domains  of 
professions  practice,  including  in  military  jobs  and  specialties. 

•  Research  needs  to  rely  on  established  theories  of  expertise  and  its  development,  including 
Cognitive  Flexibility  Theory  and  Cognitive  Transformation  Theory,  that  explain  how 
people  acquire  advanced  knowledge,  skill,  ability  to  refine  their  mental  models,  and 
ability  to  adapt  to  tough  cases. 

•  We  need  to  be  able  to  fine-tune  training  materials  and  strategies  to  the  different  levels  of 
proficiency:  The  gap  from  initiate  to  senior  apprentice,  the  gap  from  apprentice  to 
journeyman,  and  the  gap  from  journeyman  to  expert. 

•  Research  needs  to  rely  upon,  but  also  expand  upon  our  knowledge  of  what  makes  for  a 
good  mentor,  and  the  most  efficient  and  appropriate  ways  of  using  mentoring  to  achieve 
accelerated  learning.  We  need  to  expand  our  empirical  base  on  how  to  identify 
individuals  who  could  become  good  mentors,  and  how  to  teach  people  how  to  be 
mentors. 

•  Research  needs  to  rely  the  known  about  the  features  of  scenarios  that  facilitate  training, 
such  as  fidelity  and  pertinence  to  expert-level  knowledge  and  reasoning. 

The  Working  Meeting  resulted  in  a  list  of  key  considerations  in  the  design  of  a  research  and 
technology  roadmap  for  establishing  methods  of  facilitated  retention: 

•  We  need  to  expand  our  empirical  base  about  individual  difference  predictors  of  retention, 
such  as  age,  gender,  cognitive  ability  level,  education  level,  etc. 

•  We  need  to  expand  our  empirical  base  about  how  periodic  booster  sessions  (number, 
spacing)  might  affect  forgetting  rates. 

•  We  need  to  expand  our  empirical  base  about  differential  decay  of  particular  abilities. 
Booster  training  should  focus  on  refresh  of  skills  that  would  ordinarily  show  faster  drop¬ 
off. 

•  We  need  to  institute  procedures  for  rapidized  updating  both  during  and  post-hiatus, 
because  during  a  hiatus,  the  missions  and  the  technology  are  constantly  changing. 

Finally,  the  Working  Meeting  identified  a  number  of  key  science  and  technology  challenges  that 
currently  are  either  not  funded  or  are  underfunded: 

•  Understanding  the  requirements  for  expertise  in  military  jobs  and  how  to  accelerate  its 
attainment. 

•  Understanding  how  to  increase  retention  of  competence  and  proficiency. 

•  Developing  materials  and  methods  for  accelerated  learning. 

•  Developing  technology  for  rapid  understanding  of  learning  and  training  problems  (i.e., 
rapidized  cognitive  task  analysis. 

•  Determining  level  of  competence  required  for  different  stages  of  a  war  fighter's  career. 

•  Determining  effective  ways  to  compress  meaningful  learning  experiences  with 
challenging  cases. 

•  Calculating  the  cost-benefit  of  implementing  accelerated  learning. 
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IMPLICATIONS  FOR  RESEARCH  AND  ROADMAP 

The  opportunity  and  need  is  to  actually  track  changes  in  job  requirements  longitudinally  in  one 
or  more  selected  domains  where  there  is  a  rapidly  changing  job  environment  (e.g.,  technology 
adoption  issues).  This  would  be  especially  pertinent  with  regard  to  the  aging  civilian  population. 
Once  domains  of  study  are  selected,  there  must  commence  immediately  an  effort  to  forge  an 
appropriate  proficiency  scale  and  develop  methods  and  measures  for  evaluating  performance.  A 
retrospective  study  should  be  conducted  that  identifies  senior  (or  retired)  experts,  engages  then  in 
detailed  career  analysis,  and  captures  their  knowledge  and  reasoning  skill.  This  Cognitive  task 
analysis  will  result  in  work  models  and  will  support  the  process  of  creating  a  "library"  of  cases 
(problems  scenarios,  etc.)  that  capture  expert  reasoning  and  that  can  be  used  in  training. 

Research  Design 

Tracking  performance  over  the  ten-year  span  going  from  initiate  to  expert  would  be  impractical. 
It  would  be  a  major  project  to  conduct  such  study  long  enough  (more  than  five  years, 
realistically)  to  cover  hiatus  periods  over  actual  careers.  Nevertheless,  it  is  precisely  such  study 
that  would  provide  the  necessary  empirical  base.  Thus,  a  cohort-select  design  is  recommended. 
In  such  a  design,  groups  at  different  stages  of  proficiency  are  identified  and  each  is  tracked 
across  their  development  as  they  progress  to  the  next  proficiency  level. 

Acceleration  Methods 


Rapidized  training  and  accelerated  proficiency  can  be  possible  through  the  use  of  computer 
games,  simulations,  and  immersions,  all  of  which  are  enhanced  through  application  of  intelligent 
tutoring  and  artificial  intelligence  capabilities. 

Accelerated  proficiency  can  be  achieved  through  the  use  of  case-based  instruction  and  realistic 
tough  cases  with  focus  on  errors  and  "desirable  difficulties."  Expertise  is  the  source  for  training 
materials.  Training  for  accelerated  proficiency  must  rely  upon  meaningful,  corrective  feedback 
that  is  appropriately  timed  (neither  too  close  nor  too  distance  from  the  performance  being 
evaluated). 

Eor  acceleration  of  team  proficiency,  training  exercises  must  encourage  the  acquisition  of 
teamwork  skills  and  “people  skills.”  Training  must  take  into  account  individual  differences  in 
cognitive  style  and  intrinsic  motivation. 

Accelerated  proficiency  and  rapidized  training  could  be  achieved  through  "tough  case  time 
compression."  Given  that  it  takes  extensive  practice  at  difficult  problems  to  achieve  expertise, 
and  given  that  tough  cases  are  rare,  a  library  of  tough  cases  would  make  time  compression 
possible.  (This  is  not  unlike  the  method  of  the  Think  Eike  a  Commander  program.)  In  addition  to 
scenario  and  simulation-based  training,  additional  methods  for  rapidization  include  Decision 
Making  Exercises  and  Operational  Simulations,  both  of  which  are  specifically  suited  to  training 
team  decision  making. 


14 


Report  on  FA8650-09-2-6033 


Together,  the  prospective  and  retrospective  studies  will  allow  coverage  of  the  developmental 
progression  greater  than  the  five  to  eight  year  duration  of  the  actual  research  project  itself.  They 
will  feed  into  the  formation  of  general  longitudinal  models.  They  will  support  an  activity  to 
calculate  the  costs  and  benefits  of  instituting  accelerated  learning  methods,  and  the  risks  of  not 
doing  so. 

Additional  Considerations 


This  Report  identifies  some  general  requirements  for  a  research  program,  some  specific 
requirements  in  domain  choice,  and  some  methodological  and  measurement  details  that  have  to 
be  refined  and  specified:  better  measures  of  overlearning,  advances  in  scenario  engineering 
technology,  improved  computational  models  of  competence  and  proficiency,  and  means  for 
measuring  adaptation  and  resilience  capacity).  The  Report  also  lists  a  number  of  opportunities  to 
study  training  and  hiatus  effects,  including  studies  with  the  U.S.  Army  Reserves,  the  Corps  of 
Engineers,  and  the  electric  utilities.  The  paramount  goals  of  a  research  program  would  be  (1) 
Facilitating  the  achievement  of  proficiency,  especially  in  the  Apprentice  to  Senior  Journeyman 
levels,  (2)  Retaining  expertise  in  the  form  of  both  personnel  capabilities  and  in  the  form  of 
organizational  knowledge,  and  (3)  Result  in  applications  to  military  domains  including  USAF 
mission-critical  specializations. 
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1.0  INTRODUCTION 
1.1  Background 

In  recent  years  there  has  been  wide  recognition  in  the  business  community  of  the  importance  of 
knowledge  capture,  preservation,  and  sharing  in  knowledge-based  organizations.  Largely  this  is 
in  response  to  the  coming  "grey  tsunami" — the  immanent  retirement  of  senior  experts  in  business 
and  government  (see  Hoffman  &  Hanes,  2003;  Moon,  Hoffman  &  Ziebell,  2009).  A  number  of 
recent  books,  both  edited  and  authored,  both  academic  and  popular  press,  have  discussed  the 
training  issues  and  workforce  challenges  that  have  emerged  as  organizations  have  become  more 
"knowledge-based"  and  technologies  more  pervasive  in  shaping  complex  cognitive  work  (e.g., 
Davenport  &  Prusak,  1998;  Ericsson,  2009;  Goldstein,  &  Ford,  2001;  Kraiger,  2001;  Nonaka,  & 
Takeuchi,  1995;  O'Dell  &  Grayson,  1998;  Quinones  &  Ehrenstein,  1996).  The  modern 
workplace  has  been  dubbed  “sociotechnical”  in  recognition  of  the  fact  that  the  work  involves 
collaborative  mixes  of  multiple  people  and  multiple  machines.  Furthermore,  the  work  is 
generally  “cognitive  work”  (Hoffman  &  Militello,  2008).  This  characterizes  many  military  work 
settings  and  missions. 

Domain  practitioners  who  achieve  high  levels  of  proficiency  provide  technical  judgment  to  speed 
decision-making  in  time-critical  events.  They  provide  resilience  to  operations  by  resolving  tough 
problems,  anticipating  future  demands  and  re-planning,  and  acting  prudently  by  judgment  rather 
than  by  rule.  High  proficiency  practitioners  exercise  effective  technical  leadership  in  ambiguous 
or  complex  situations,  often  by  communicating  subtle  features  that  other  people  will  not  see  until 
they  are  pointed  out.  It  typically  takes  years  of  experience  for  professionals  to  master  their 
domain.  Reasons  for  this  include  domain  complexity  and  the  need  for  extended  and  continuing 
practice  at  tough  cases. 

As  workplaces  and  jobs  become  more  cognition-intensive,  organizations  need  to  take  traditional 
notions  of  training  to  new  levels,  and  well  into  the  territory  of  complex  systems.  The  workers  in 
sociotechnical  systems  must  be  trained  to  be  adaptive,  so  that  they  can  cope  with  the  ever- 
changing  world  and  ever-changing  workplace.  People  must  be  trained  to  be  resilient,  so  that  they 
can  cope  with  complexity  when  unexpected  events  stretch  resources  and  capabilities.  And 
workers  must  be  trained  faster.  Intelligent  systems  technology,  and  intelligent  use  of  technology, 
will  certainly  play  a  critical  role  in  this. 

Training  for  the  achievement  of  expertise  has  become  a  salient  topic  for  discussion  at  research 
and  technology  meetings  sponsored  by  all  branches  of  the  military  (e.g.,  Hszieh,  Shobe  & 
Wulfleck,  2009).  Many  current  military  jobs  (estimated  at  85%)  can  be  trained  through 
established  methods,  and  those  jobs  involve  tasks  that  can  be  performed  by  individuals  who  are 
proficient  (Wulfleck  &  Wetzel-Smith,  2008).  In  classical  guild  terminology,  they  would  be 
"journeymen" — they  have  practiced  to  the  point  where  they  can  perform  their  duties 
unsupervised  (literally,  they  can  go  on  a  journey).  The  military  has  need  for  personnel  who  are 
trained  at  a  number  of  levels  of  proficiency.  While  there  may  be  some  need  for  more  Senior 
Experts  in  select  areas,  there  is  a  definite  and  continuing  need  across  the  military  for  Journeymen 
and  Senior  Journeymen. 
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One  reason  is  the  constantly-changing  nature  of  the  sorts  of  missions  that  the  military  has  to 
conduct  and  the  various  jobs  involved.  Furthermore,  jobs  must  be  adaptive  to  constantly 
changing  threats  or  circumstances.  In  a  sense,  everything  is  getting  more  complex  and  important 
work  is  often  cognitive  work  (Wulfleck  &  Wetzel-Smith,  2008).  Research  in  the  field  of 
Expertise  Studies  has  shown  over  and  over,  in  diverse  domains  from  medicine  to  firefighting, 
that  it  takes  years  of  deliberate  practice  for  individuals  to  master  their  domain,  and  this  finding 
applies  for  many  military  jobs  (e.g.,  anti-submarine  warfare).  Reasons  why  extended  training  and 
experience  are  required  to  achieve  high  levels  of  proficiency  include  domain  complexity  and  the 
need  for  practice  at  tough  cases. 

Across  recent  decades,  many  military  workplaces  have  changed,  and  many  new  ones  emerged,  as 
forms  of  "complex  sociotechnical  systems"  in  which  the  work  is  cognitive  and  collaborative,  and 
heavily  reliant  on  computer  technology.  Work  in  such  domains  requires  high  levels  of 
proficiency,  in  terms  of  knowledge,  reasoning  skill,  and  critical  thinking  skill.  Specific  domains 
include  command  posts,  intelligence  analysis,  emergency  response,  disaster  relief,  and 
cyberdefense.  Within  the  military,  the  required  preparedness  status  includes  capabilities  to  be 
adaptive,  resilient,  and  robust  in  the  face  of  unexpected  disruptions.  This  is  referred  to  as 
"cognitive  readiness"  (Morrison  &  Fletcher,  2002). 

Many  current  military  career  paths  involve  training  to  proficiency  (often,  high  levels  of 
proficiency)  and  then  reassignment  for  some  period  of  time  (sometimes  three  or  more  years)  at 
some  other  job.  The  classic  example  is  that  of  pilot,  trained  to  proficiency  and  tested  in  combat, 
and  then  assigned  to  duty  at  the  Pentagon.  Temporary  reassignment  is  commonplace,  and  raises 
additional  issues  in  the  general  area  of  training,  specifically  issues  of  transfer,  decay,  and 
retention  of  knowledge  and  .skill. 

Many  organizations  (e.g.,  DoD,  NASA,  the  electric  utilities)  are  at  risk  because  of  the  imminent 
retirement  of  domain  practitioners  who  are  relied  upon  to  handle  the  most  difficult  and  mission- 
critical  challenges  (Hoffman  &  Hanes,  2003).  This  Report  deals  with  current  issues  of  training: 

•  How  to  quicken  the  training  process  while  maintaining  its 
effectiveness  (Rapidized  Training), 

•  How  to  rapidize  the  transposition  of  lessons  learned  from 
the  battlespace  into  the  training  context  {Rapidized 
Knowledge  Sharing), 

•  How  to  train  and  train  quickly  to  higher  levels  of 
proficiency  (Accelerated  Proficiency),  and 

•  How  to  insure  that  training  has  a  stable  and  lasting  effect 
(Facilitated  Retention). 

These  four  concepts  refer  to  different  workforce  issues,  and  have  different  implications  for 
training  research  and  training  methodology.  They  are  the  meanings  of  accelerated  learning  are 
what  we  will  refer  to  in  this  Report. 
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1.2  The  Workshops  on  Accelerated  Learning 

A  "Workshop  on  Accelerated  Learning"  (June,  2008)  was  sponsored  by  the  Human  Effectiveness 
Directorate  and  the  DoD  Accelerated  Learning  Technology  Focus  Team.  The  Workshop  brought 
together  leading  academic,  private  sector,  and  DoD  specialists  in  areas  of  training  and  expertise 
studies,  to  discuss  the  notion  of  accelerated  learning.  Presentations  at  that  Workshop  laid  some  of 
the  many  challenges  and  issues  for  accelerated  learning. 

A  "Working  Meeting  on  Accelerated  Proficiency  and  Facilitated  Retention"  (October,  2009)  was 
sponsored  by  the  711  Human  Performance  Wing,  Human  Effectiveness  Directorate,  Air  Force 
Research  Laboratory.  An  interdisciplinary  group  of  participants  was  presented  the  specific  task 
of  specifying  methods  and  research  ideas  for  attempts  to  accelerate  the  proficiency  and  facilitate 
the  retention  of  expertise  in  selected  mission-critical  domains.  The  participants  were  to  address 
the  following  questions: 

•  How  can  we  develop  operational  definitions  and  measures  of  proficiency  at  a  fine  grain? 

•  How  can  we  develop  methods  for  identifying  expert  mentors  and  revealing  their 
knowledge  and  strategies? 

•  How  can  we  best  design  training  to  promote  skill  retention  and  prevent  skill  decay  during 
periods  of  hiatus? 

•  How  can  we  train  for  adaptivity  and  the  need  to  cope  with  the  ever-changing  workplace 
and  changing  and  challenging  missions? 

•  How  can  we  train  for  resilience  and  the  need  to  cope  with  complexity  when  unexpected 
events  stretch  resources  and  capabilities? 

•  What  USAF  jobs  require  high  levels  of  proficiency  and  have  analogs  in  the  private 
sector? 

•  How  can  we  construct  optimal  strategies  for  mitigating  at  least  some  skill  decay  while 
warfighters  are  on  a  long  term  hiatus.” 

Responses  to  these  questions  would  feed  into  the  formulation  of  guidance  for  attempts  to 
accelerate  the  achievement  of  proficiency. 

This  report  integrates  the  following: 

(1) .  The  results  of  this  second  Workshop, 

(2) .  A  literature  review  of  research  and  ideas  on  training,  skill  decay,  and  transfer, 

specifically  with  respect  to  concepts  of  acceleration,  and 

(3) .  Recommendations  and  suggestions  for  research  and  applications  activities. 

1.3  The  Challenges 

The  phrase  "accelerated  learning"  became  common  currency  over  a  span  of  just  a  couple  of 
years,  culminating  in  the  popularization  of  this  scientific  concept.  Indeed,  there  is  now  a  web  site 
that  promotes  “Seven  principles  of  accelerated  learning.” 

[http://www.discoverylearning.co.uk/principles/index.html].  This  appears  to  be  a  course  offering 
that  is  built  upon  some  simple  generalizations  about  memory  and  the  brain.  The  only  question  we 
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might  ask  is,  What  do  the  course  providers  mean  by  accelerated  learning?  Lacking  some  analysis 
of  retention/execution  across  a  proficiency  continuum,  the  claim  would  be  that  the  course  teaches 
about  "learning." 

Acceleration  can  always  be  achieved  by  improving  training  methods  that  are  clearly  lacking  in 
structure  or  effectiveness.  For  instance,  many  organizations  proclaim  the  value  of  “on-the-job” 
training  (Derouin,  Parrish,  &  Salas,  2005),  but  have  no  structured  focus  for  the  learning, 
inadequate  plans  and  procedures,  and  inadequate  management  or  organizational  support 
(Stanard,  et  ah,  2002).  Acceleration  in  the  sense  of  improving  deficient  training  is  not  the  focus 
of  this  Report. 

Accelerated  Learning  has  been  defined  in  a  number  of  ways  (Hoffman,  et  ah,  2009),  and  we 
focus  on  three  of  them,  which  we  listed  above.  The  first  is  "rapidized  training" — the  idea  of 
training  individuals  to  achieve  some  minimal  level  of  proficiency  at  a  rate  faster  than  that 
achieved  by  the  current  or  traditional  training.  Second,  accelerated  learning  also  refers  to  the  idea 
of  getting  individuals  to  achieve  high  levels  of  proficiency  at  a  rate  faster  than  ordinary.  This  is 
the  second  meaning  that  we  listed  above.  The  most  succinct  way  of  saying  this  is  to  ask:  "Can  we 
turn  an  apprentice  into  an  expert  in  less  than  ten  years?"  Third,  accelerated  learning  also  refers  to 
the  idea  of  making  learning  more  immune  to  decay.  Once  trained  to  a  high  level  of  proficiency, 
how  can  one  stabilize  that  level  of  .skill? 

We  take  each  of  these  senses  of  “accelerated  learning”  as  a  challenge. 

Challenge  1:  Accelerated  Learning  in  the  Sense  of  Rapidized  Training 

The  goal  of  improving  training  is  always  paramount  for  trainers,  instructional  designers,  and 
human  resources  specialists.  There  is  always  a  search  for  better  methods  or  for  improvements  of 
existing  methods.  Entire  organizations  within  the  U.S.  Military,  and  not  just  individual  programs, 
have  the  goal  of  improving  training  through  innovations  such  as  distance  learning  and 
simulation.  Each  branch  of  the  military  has  Commands  for  training,  such  as  the  Air  Eorce’s  Air 
Education  and  Training  Command,  Army’s  Training  and  Doctrine  Command,  and  the  Navy’s 
Naval  Education  and  Training  Command. 

Recent  years  have  witnessed  a  significant  diversification  of  critical  missions  that  the  military 
must  perform,  shifting  from  traditional  combat  to  counter-insurgency,  emergency  relief,  and 
human  terrain.  New  capabilities,  such  as  Unmanned  Vehicles,  have  changed  doctrine  and 
operations  across  a  broad  spectrum  of  activities.  The  pace  of  change,  and  not  just  change  itself, 
presents  new  and  significant  challenges,  which  manifest  as  issues  for  training  and  workforce 
capabilities. 

Rapidized  training  is  the  goal  of  completing  a  training  program  in  less  time  than  ordinarily  taken, 
while  achieving  the  same  levels  of  performance  or  proficiency  at  the  end  of  the  training.  A  more 
stringent  definition  would  include  the  additional  goal  of  maintaining  the  same  level  retention 
(performance  tested  some  time  after  the  conclusion  of  the  training)  or  even  resulting  in  improved 
retention. 
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Rapidized  training  also  includes  the  idea  of  speeding  up  the  process  of  assimilating  lessons 
learned,  converting  them  into  instructional  materials,  and  applying  them  in  the  battlefield 
context.  This  goal  is  significant  given  the  rapid  pace  at  which  adversaries  can  adapt. 

Challenge  2:  Accelerated  Learning  in  the  Sense  of  Accelerated  Proficiency 

"Accelerated  learning"  refers  not  only  to  the  idea  of  hastening  the  achievement  of  basic-level 
proficiency;  it  reaches  across  the  proficiency  scale  to  the  question  of  how  to  accelerate  the 
achievement  of  expertise,  and  whether  that  is  even  possible.  It  is  widely  recognized  that 
warfighters  and  teams  must  be  trained  to  high  levels  of  proficiency,  even  the  level  of  expertise, 
because  of  the  need  for  operations  that  rely  on  cognitive  work  and  are  robust,  resilient,  and 
adaptive. 

•  Robustness  is  the  ability  to  maintain  effectiveness  across  a  range  of  tasks,  situations,  and 
conditions. 

•  Resilience  is  the  ability  to  recover  from  a  destabilizing  perturbation  in  the  work  as  it 
attempts  to  reach  its  primary  goals. 

•  Adaptivity  is  the  ability  to  employ  multiple  ways  to  succeed  and  the  capacity  to  move 
seamlessly  among  them. 

Experts  and  Senior  Journeymen  are  individuals  who  achieve  high  levels  of  proficiency.  Such 
individuals  provide  technical  judgment  to  speed  decision-making  in  time-critical  events.  They 
provide  resilience  to  operations  by  resolving  tough  problems,  anticipating  future  demands  and 
re-planning,  and  acting  prudently  by  judgment  rather  than  by  rule.  High-proficiency  practitioners 
exercise  effective  technical  leadership  in  ambiguous  or  complex  situations,  often  by 
communicating  subtle  features  that  other  people  will  not  see  until  they  are  pointed  out. 

It  typically  takes  years  of  experience  for  professionals  to  master  their  domain.  Reasons  for  this 
include  domain  complexity  and  the  need  for  extended  and  continuing  practice  at  tough  cases. 
Fundamental  to  the  achievement  of  robustness,  resilience  and  adaptivity  is  the  opportunity  for 
practice  at  problems  that  stretch  current  competency  (Feltovich  et  ah,  1997).  Professionals  must 
acquire  knowledge  and  reasoning  skills  that  pertain  to  critical  domain  goals  but  which  must  be 
exercised  in  differing  situations  or  contexts.  Capability  must  transfer  in  this  sense. 

Challenge  3:  Accelerated  Learning  in  the  Sense  of  Facilitated  Retention 

The  challenge  of  achieving  and  maintaining  high  levels  of  proficiency  is  compounded  in  the 
military  by  such  practices  as  collateral  assignment,  redeployment  (e.g.,  rapid  skill  decay  on  the 
part  of  pilots  creates  a  need  for  expensive  re-training),  inadequate  or  ad  hoc  mentoring,  and  the 
drive  for  just-in-time  training.  Another  significant  challenge  is  clustered  around  career  (versus 
job)  training,  and  retention.  Indeed,  the  entire  field  of  “knowledge  management”  is  formed 
around  the  notion  of  achieving  stability. 

Skill  decay  is  particularly  salient  and  problematic  in  situations 
where  individuals  receive  initial  training  on  knowledge  and  skills 
that  they  may  not  be  required  to  use  or  exercise  for  extended 
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periods  of  time.  Reserve  personnel  in  the  military,  for  example,  may 
be  provided  formal  training  only  once  or  twice  a  year.  When  called 
up  for  active  duty,  however,  it  is  expected  that  they  will  need  only  a 
limited  amount  of  refresher  training,  if  any,  to  reacquire  any  .skill 
that  has  been  lost  and  subsequently  to  perform  their  mission 
effectively  (Arthur,  et  ah,  1998,  p.  58;  see  also  Wisher,  Sabol, 
Hillel,  &  Kern,  1991). 


1.4  The  Concept  of  Accelerated  Learning 

The  concept  of  accelerated  learning  was  referenced  by  Gott  (1995)  in  a  discussion  of  how  the 
acquisition  of  proficiency  on  the  part  of  workers  in  sociotechnical  domains  is  possible  only  after 
the  development  of  rich  mental  models  and  strategic  reasoning  skills;  that  is,  the  apprentice  who 
desires  to  achieve  high  levels  of  proficiency  has  to  engage  in  difficult  cognitive  work. 

The  concept  of  accelerated  learning  was  implicit  in  discussions  of  the  concept  of  the  “expert 
apprentice.”  The  idea  here  is  that  knowledge  management  depends  on  having  a  workforce  of 
proficient  knowledge  elicitors  who  are  trained  to  be  able  to  rapidly  achieve  the  level  of 
understanding  of  an  advanced  apprentice,  minimally.  Only  by  acquiring  domain  knowledge  at 
that  level  can  they  contribute  substantively  to  processes  of  capturing  and  preserving  expert 
knowledge  (Militello  &  Quill,  2007).  The  field  of  Knowledge  Management  has  a  theme  of 
accelerated  learning,  which  is  not  surprising  given  the  business  incentives  to  train  faster  and 
better.  One  goal,  for  instance,  is  to  reduce  “time  to  value”  in  product  innovation.  Indeed,  the  field 
of  Knowledge  Management  focused  on  issues  of  learning  and  training.  This  is  shown  by  the 
emergence  of  the  roles  of  Chief  Knowledge  Officer  and  Chief  Learning  Officer,  and  is  reflected 
in  magazine  articles  having  titles  such  as  “Learning  at  Top  Speed”  (Atkinson  &  Barry,  2010). 

The  early  work  on  “expert  systems”  led  to  the  vision  that  organizations  might  create  large 
knowledge  repositories  (Becerra-Fernandez  &  Leidner,  2008).  Knowledge  Management  software 
systems  differ  from  traditional  management  information  systems  in  that  Knowledge 
Management  software  tools  help  create  the  very  content  on  which  they  operate.  Like  traditional 
management  information  systems,  however,  there  are  issues  of  acceptance  and  integration  into 
business  procedures  and  organizational  cultures. 

Over  the  years  since  Gary  Klein’s  seminal  publication  on  "Preserving  Corporate  Knowledge" 
(1992),  numerous  articles  and  trade  books  have  appeared  bearing  such  titles  as:  T/'we  only  knew 
what  we  know  (O'Dell  &  Grayson,  1998)  and  The  knowledge  creating  company  (Nonaka  & 
Takeuchi,  1995)  (see  also  Alice,  1997;  Brooking,  1999;  Choo,  1988;  Davenport  &  Prusak,  1998; 
Lambe,  2007;  Leonard  &  Swap,  2005).  All  of  these  discuss  expertise  (or  "core  competencies"), 
knowledge  elicitation  and  knowledge  repositories.  These  books  illustrate  what  some  see  as  the 
knowledge  management  craze  of  the  late  1990s,  when  upwards  of  25  percent  of  Fortune  500 
companies  had  a  Corporate  Knowledge  Office  (Pringle,  2003).  Organizations  such  as  IBM  and 
the  World  Bank  have  made  substantial  investments  in  support  of  organizational  knowledge 
capture  and  management.  Norman  Kamilkow,  Editor  of  Chief  Learning  Officer  Magazine  said. 
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What  we  saw  was  that  there  is  a  growing  role  for  a  chief  learning 
officer  type  within  enterprise-level  companies  .  .  .  there  is  a  need  to 
have  somebody  focused  on  how  to  keep  the  skills  of  the 
corporation's  work  force  at  a  high  level  (quoted  in  Pringle,  2003,  p. 

Bl). 

In  the  Knowledge  Management  process,  company  management  establishes  a  program  whereby 
experts  who  possess  valuable  undocumented  knowledge  collaborate  with  a  knowledge  engineer. 
Working  together,  they  elicit  the  worker’s  wisdom  for  inclusion  in  the  organization’s  knowledge 
base.  In  extreme  cases,  such  as  a  senior  worker  retiring,  the  individual  might  be  retained  or 
brought  back  as  a  consultant  (Becerra-Fernandez  &  Leidner,  2008). 

The  field  of  Knowledge  Management  raises  the  practical  problem  of  knowledge  finding: 
Identifying  individuals  who  possess  knowledge  that  is: 

1 .  Unique  to  them, 

2.  Critical  to  the  organization,  and 

3.  Tacit  in  the  sense  of  being  undocumented. 

This  has  been  recognized  as  a  key  to  the  success  of  Knowledge  Management  broadly  (Gaines, 
2003;  Gross,  Hanes,  &  Ayres,  2002;  Hanes  &  Gross,  2002).  Recent  experience  shows  that  is  it 
possible  and  sometimes  fairly  easy  for  experts  and  managers,  working  together,  to  identify  the 
unique  and  important  knowledge  areas  in  which  a  particular  expert  excels.  Likewise,  domain 
practitioners  can  readily  identify  those  important  concepts  in  a  domain  that  seem  to  be  especially 
difficult  for  others  to  fully  comprehend  (Dawson-Saunders,  et  ah,  1990).  A  critical  gap,  however, 
is  that  a  robust,  general  procedure  for  doing  this  has  not  been  formulated  in  such  a  way  that 
anyone  might  implement  it. 

Knowledge-intensive  organizations  rely  on  decision-makers  to  produce  mission  critical  decisions 
based  on  inputs  from  multiple  domains  (Becerra-Fernandez,  et  ah,  2004).  The  decision-maker 
needs  an  understanding  of  many  specific  sub-domains  that  influence  the  decision-making 
process,  coupled  with  the  experience  that  allows  quick  and  decisive  action  based  on  such 
information  (Nonaka  &  Takeuchi,  1995). 

An  additional  recent  awareness  is  that  knowledge  management  via  knowledge  capture  and 
knowledge  repositories  is  only  a  part  of  the  solution  to  workforce  problems. 

If  an  organization  could  capture  the  knowledge  embedded  in  clever 
people's  minds,  all  it  would  need  is  a  better  knowledge-management 
system.  The  failure  of  such  systems  to  capture  tacit  knowledge  is 
one  of  the  greatest  disappointments  of  knowledge-management 
initiatives  to  date  (Goffee  &  Jones,  2007). 

What  is  needed  are  new  approaches  to  knowledge  training,  in  particular,  a  method  for 
accelerating  the  achievement  of  high  levels  of  proficiency.  Recognition  of  this  need  is  illustrated 
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by  the  many  recent  books  that  present  methods  for  training  an  expert  business  workforce  (e.g., 
Clark,  2008;  Goldstein  &  Ford,  2002;  Kraiger,  2002;  Quinones  &  Ehrenstein,  1997). 

The  Air  Force  Weather  Agency  has  launched  a  program  to  train  "Forecast  Ready"  forecasters 
possessing  high-level  knowledge  beyond  intermediate  skill  levels.  Such  forecasters  are  capable 
of  using  mesoscale  models  and  remote  sensing  tools.  Such  forecasters  are  able  to  "explain  the 
reasoning  behind  the  forecast"  (McNulty,  2005,  p.  5).  To  make  it  possible  for  training  to 
accelerate  learning  in  this  sense,  great  reliance  will  be  placed  on  mechanisms  of  distance 
learning  (Pagitt,  2005).  "The  7th  Weather  Squadron  is  moving  at  lightning  speed  towards  a  new 
training  initiative...  a  premier,  just-in-time  combat  field  skills  training  course..."  (7th  Weather 
Squadron,  2005,  p.  16). 

There  are  domains  of  expertise  and  specialization  within  the  military  that  are  especially  strategic 
and  therefore  important  from  a  workforce  and  training  perspective,  and  to  which  we  would 
ideally  be  able  to  apply  methods  of  acceleration.  An  example  domain  is  maintenance  of  strategic 
nuclear  and  non-nuclear  strike  systems.  Although  this  domain  is  not  a  specific  focus  of  the 
present  Report,  it  does  serve  to  highlight  some  issues. 

Future  needs  for  specialized  skill  sets  required  in  the  domain  of  strategic  strike  capabilities  have 
been  the  subject  of  extensive  study  (e.g..  Defense  Science  Board,  2006).  Because  of  this 
domain’s  importance,  and  some  sense  of  urgency  on  the  part  of  the  Department  of  Defense, 
forays  into  research  on  innovations  in  training  would  not  be  immediately  helpful  and  therefore 
not  prudent.  On  the  other  hand,  the  recommendations  of  the  Defense  Science  Board  speak 
directly  to  the  motivation  for  the  study  of  accelerated  learning,  including  the  immanent  problem 
of  a  retiring  generation  of  engineers  and  the  concomitant  loss  of  organizational  knowledge  and 
skill. 

Despite  the  fact  that  both  the  Navy  and  Air  Force  have  programs  designed  to  maintain  NBC 
strike  capabilities,  the  DSB  expressed  concerns  about:  (1)  the  lack  of  knowledge  management 
processes  implemented  across  all  of  the  pertinent  organization,  to  identify,  track  and  retain 
critical  engineering  skills,  (2)  the  challenge  of  attracting  the  best  students  to  the  pertinent  science 
and  engineering  disciplines,  (3)  the  need  for  high  levels  of  proficiency  in  order  to  cope  with 
unanticipated  failures  requiring  analysis  and  re-design,  and  (4)  the  need  for  a  new  generation 
trained  for  adaptation  to  new  concepts  and  emerging  technologies. 

“The  main  finding  for  our  project  is  the  perceived  dearth  of 
expertise,  current  and  future,  in  various  of  these  areas,  particularly 
in  the  area  of  ballistic  missiles.  In  addition,  there  is  the  assessment 
that  current  Human  Capital  Management  Systems"  are  insufficient 
for  identifying,  tracking,  and  managing  critical  skills”  (Defense 
Science  Board,  2006). 

These  are  the  kinds  of  challenges  to  which  accelerated  learning  should  be  applicable. 
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The  four  meanings  of  aecelerated  learning  are  captured  in  the  Concept  Map  shown  in  Figure  1 . 
Figure  2  is  a  Concept  Map  that  describes  the  related  challenges.  (This  diagram  was  included  in 
the  handout  material  for  the  Working  Meeting.) 


Figure  1 . 1 .  A  Concept  Map  about  Accelerated  Learning 
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Figure  1.2.  A  Concept  Map  about  the  goals  and  challenges  of  Accelerated  Learning. 


26 


Report  on  FA8650-09-2-6033 


2.0  LITERATURE  REVIEW 

2.1  Introduction  to  the  Literature  Review 


Scientific  literatures  that  are  pertinent  to  accelerated  learning  and  facilitated  retention  can  be 
organized  and  summarized  in  a  number  of  ways.  One  approach  is  to  consider  studies  of  skill 
versus  knowledge.  Another  approach  is  to  consider  studies  of  long-term  retention  versus  decay. 
One  can  consider  studies  of  training  in  general,  and  military  training  in  particular.  One  can 
consider  studies  of  individual  proficient  performers  or  studies  of  teams.  Cutting  across  all  of  the 
above,  one  can  consider  issues  of  measurement  and  evaluation. 

We  begin  with  a  consideration  of  "types"  of  knowledge.  This  topic  raises  many  scientific  and 
philosophical  issues,  but  especially  issues  regarding  the  acquisition  and  elicitation  of  knowledge 
(Hoffman  &  Militello,  2008;  Carlson  2008).  A  distinction  that  is  widely  drawn  in  the  literature  of 
the  learning  sciences  is  that  between  declarative  and  procedural  knowledge  (as  in  Anderson, 
2000).  This  distinction  is  slippery  (concept  classification  an  be  regarded  as  a  skill).  There  is  a 
huge  difference  between  declarative  knowledge  in  the  sense  of  remembering  terms  and  concepts 
versus  declarative  knowledge  in  the  sense  of  understanding  complex  causal  relationships.  There 
is  a  huge  difference  between  procedural  knowledge  in  the  sense  of  stable  sequences  versus 
procedural  knowledge  in  the  sense  of  learning  to  use  heuristics  to  achieve  some  goal. 

Another  distinction  is  between  “memory”  and  “skill,”  with  skill  traditionally  meaning  motor  skill 
(typified  in  the  academic  laboratory  by  the  pursuit  rotor  and  the  typewriter).  This  distinction  is 
slippery  because  memorization  is  a  skill  that  can  involve  applying  numerous  strategies.  In 
modem  sociotechnical  work  systems,  reasoning  strategies  are  considered  to  be  sets  of  leamable 
“skills”  We  find  in  this  one  example  a  need  to  be  cautious  about  conceptual  terminology  that  is 
laden  in  tradition.  Complex  cognitive  work  typically  involves  acting  and  is  not  merely  thinking. 
The  medical  sonographer  has  to  know  how  to  control  the  sensor,  the  pilot  has  to  know  how  to 
move  the  stick.  It  would  be  misleading  to  call  these  “motor  skills”  or  “behaviors,”  since  the 
activities  are  knowledge-driven,  perception-driven  and  context-dependent. 

The  phrase  “declarative  knowledge”  assumes  a  class  of  knowledge  that  is  not  declarative  or  not 
verbalizable  in  principle.  This  latter  category  represents  an  indeterminate  class,  sometimes  called 
"tacit"  knowledge,  hiding  in  a  philosophical  hornet’s  nest.  Thus,  we  might  refer  instead  to 
Teachable/Conceptual  Knowledge  as  knowledge  that  is  developed  by  being  taught  or  mentored 
and  by  studying  training  materials.  Experiential  Knowledge  can  be  learnt  only  by  direct  personal 
experience — actually  being  there  to  know  what  it  felt  like.  Conjectural  Knowledge/Skill  is  ability 
at  strategic  sensemaking,  mental  projection,  insightfulness,  resourcefulness,  opportunism  and 
similar  capabilities. 

These  distinctions,  while  slippery,  are  important  because  all  generalizations  about  learning, 
retention,  decay,  transfer,  the  role  of  feedback,  etc.  regarding  one  type  are  not  likely  to  apply 
directly  to  other  types.  Various  combinations  and  degrees  of  the  types  are  involved  in  individual 
sub-tasks  and  are  admixed  further  in  the  composition  of  tasks  and  jobs.  The  definition  of  a 
concept  can  be  memorized  (declarative  knowledge),  it  can  be  understood  through  comparison 
and  contrast  with  related  concepts  (declarative  knowledge),  or  it  can  be  learned  by  classifying 
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examples  and  nonexamples  (procedural  knowledge).  The  same  is  true  for  principles  (e.g.  causal 
models)  and  for  procedures.  Memorization,  understanding,  and  application  are  different  kinds  of 
learning,  just  as  factual  conceptual,  procedural  (steps),  and  causal  knowledge  are  all  very 
different,  and  all  of  them  are  required  to  reach  proficiency  in  a  complex  cognitive  task 
(Reigeluth,  personal  communication). 

For  further  discussion  of  distinctions  on  types  of  knowledge  and  skill,  see  Chapter  5  in  Hoffman 
and  Militello  (2008). 

2.2  The  Nature  of  Proficiency 

Three  topics  of  proficiency  development  are  particularly  pertinent  to  accelerated  learning:  The 
nature  of  high  proficiency,  proficiency  scaling  concepts  and  methods,  and  the  features  that  make 
the  achievement  of  high  proficiency  difficult. 

2.2. 1  High  Proficiency 

In  this  Report  we  are  considering  a  concept  of  expertise  referred  to  as  “high  proficiency.”  To 
accelerate  proficiency,  one  must  accelerate  the  acquisition  of  knowledge  that  is  extensive  and 
highly  organized.  One  must  also  accelerate  the  acquisition  of  expert-level  reasoning  skills  and 
strategies  (Klein,  et  ah,  2003).  Thus,  expertise  is  the  “gold  standard”  for  training,  it  is  the  source 
for  training  materials,  and  it  is  a  goal  for  accelerated  learning. 

One  might  anchor  any  proficiency  scale  with  a  concept  of  “master.”  This  category  is  used  in 
some  modem  proficiency  scales,  such  as  that  for  chess.  Simonton,  (1990,  2004)  has  used  an 
historiographic  approach  to  study  the  progression  from  expert  to  senior  expert  to  master  in  such 
domains  as  science.  In  some  craft  guilds  of  the  middle  ages,  an  expert  was  designated  a  master 
when  he  had  created  what  was  popularly  regarded  as  a  “masterpiece.”  In  some  craft  guilds  a 
maestro  was  simply  the  guild  leader.  In  some  craft  guild  traditions,  a  master  was  an  expert  who 
was  elected  to  the  council  that  set  standards.  In  yet  other  craft  guilds  a  “master”  was  any 
practitioner  who  was  qualified  to  have  apprentices.  (That  is,  qualified  specifically  to  be  a  mentor; 
a  journeyman  could  be  a  master).  Because  of  this  polysemy,  and  linkage  to  considerations  other 
than  domain  proficiency  per  se,  we  chose  not  to  rely  on  the  concept  of  master. 

Research  in  the  field  of  Expertise  Studies  has  burgeoned  since  its  inception  in  the  mid  to  late 
1980s  (e.g.,  Chi,  2005;  Ericsson,  et  ah,  1981;  Hoffman,  1992).  (Eor  reviews  see  Bereiter  & 
Scardamalia,  1993;  Ericsson,  et  ah,  2005;  Eeltovich,  et  ah,  1997). 

The  expert  is  a  repository  of  vast  historical  information  that  enables  the  exercise  of  effective 
technical  leadership  in  ambiguous  or  complex  situations,  often  by  communicating  subtle  features 
that  other  people  will  not  see  until  they  are  pointed  out.  Estimates  of  the  extent  of  knowledge  of 
the  average  person  put  it  at  anywhere  up  to  one  million  concepts  (or  "frames")  (e.g.,  four  new 
long-term  memories  per  hour  over  a  lifetime)  (Eenat  &  Eeignebaum,  1987).  Estimates  of  the 
extent  of  expert  knowledge  put  it  anywhere  from  tens  to  hundreds  of  thousands  of  individual 
propositions  (Eenat  &  Eeigenbaum,  1987),  and  somewhere  in  the  neighborhood  of  50,000 
concepts  (or  "chunks")  (e.g.,  chess  masters  can  purportedly  recognize  tens  of  thousands  of  game 
patterns)  (Reddy,  1988;  Simon  &  Barenfeld,  1969;  Simon  &  Gilmartin,  1973).  A  review  of  the 
expert  systems  literature  shows  that  some  prototype  systems  have  as  few  as  50  rules.  However, 
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as  many  as  2,000  rules  are  needed  even  for  tasks  that  are  apparently  simple,  such  as  landuse 
classification.  In  expressing  their  pessimisms  about  the  goals  of  AI,  Dreyfus  and  Dreyfus  (1986) 
asserted  that  simply  "too  many"  propositions  or  frames  would  be  needed  to  represent  even  a 
limited  domain  of  common  sense  knowledge,  let  alone  expert  knowledge. 

Expert  knowledge  is  certainly  extensive,  and  can  be  so  extensive  that  even  people  who  are 
regarded  as  "novices"  can  actually  be  very  highly  experienced.  In  the  judging  of  livestock,  for 
example,  one  might  remain  "novice"  even  after  as  much  as  ten  years  of  experience  at  school  and 
club  training.  Generally,  those  who  are  regarded  as  "experts"  have  20  to  30  years  of  experience 
in  domains  such  as  livestock  judging  and  auditing  studied  by  James  Shanteau  (1988).  Renowned 
expert  aerial  photo  interpreters  have  decades  of  experience  behind  them  (Hoffman,  1984). 
Workers  in  such  domains  naturally  distinguish  their  peers  who  are  good,  even  very  very  good, 
from  the  "real"  experts. 

Proficiency  is  defined  not  just  in  terms  of  knowledge  but  also  in  terms  of  reasoning  strategies 
and  skills.  In  one  classis  study,  Larkin  (1983)  asked  physics  students  (we  would  class  them  as 
initiates  or  apprentices)  and  experienced  physicists  to  solve  mechanics  problems  involving 
levers,  weights,  inclined  planes,  pulleys,  forces,  etc.  The  results  of  a  number  of  such  studies  have 
shown  that  the  basic  reasoning  operations  or  strategies  (deduction,  induction,  goal 
decomposition,  etc.)  are  applied  in  different  orders  and  with  different  emphases  for  experts 
versus  non-experts.  In  the  initial  stages  of  problem  solving,  experts  spend  proportionately  more 
time  than  students  in  forming  a  conceptual  understanding  of  the  problem.  Experts  generate 
representations  that  are  conceptually  richer  and  more  organized  than  those  of  the  students. 
Students  tend  to  use  hastily-formed  "concrete"  (that  is,  superficial)  problem  representations 
whereas  experts  use  "abstract"  representations  that  rely  on  "deep"  knowledge,”  that  is,  imaginal 
and  conceptual  understanding  of  functional  relations  and  physical  principles  that  relate  concepts 
(in  the  case  of  the  research  on  experts  at  mechanics,  principles  such  as  conservation  of  energy). 
Eurthermore,  experts  are  better  able  to  gauge  the  difficulty  of  problems  and  know  the  conditions 
for  the  use  of  particular  knowledge  and  procedures  (for  example,  if  there  is  acceleration,  use 
Newton's  second  law)  (Chi  et  ah,  1982). 

One  widely  cited  rule  of  thumb  is  that  the  development  of  high  proficiency  takes  at  least  ten 
years  (Hayes,  1985).  Another  rule  of  thumb  is  10,000  hours  of  practice  or  engagement  with  a 
task.  (Ten  years  at  a  full  time  job  would  be  about  20,000  hours,  assuming  that  in  an  eight  hour 
work  day  the  typical  worker  spends  about  half  their  time  engaged  in  their  fundamental  task 
work.)  These  rules  are  admittedly  approximate  and  there  is  considerable  variation  across 
domains  (Hoffman  &  Militello,  2009). 

It  is  obvious  that  mere  time  in  grade  does  not  enable  just  anyone  to  qualify  as  an  expert.  A 
person  may  achieve  journeyman  status,  become  demotivated,  and  stay  at  that  level  for  the  rest  of 
their  career.  To  achieve  high  proficiency  in  the  military  there  needs  to  be: 

•  Critical  skills  for  accomplishing  the  mission  and  achieving  national  defense, 

•  A  constant  'stretching'  of  the  skill,  defined  by  increasing  challenges  (tough  or  rare  cases), 

•  High  levels  of  intrinsic  motivation  to  work  hard,  on  hard  problems, 

•  Practice  that  provides  rich,  meaningful  feedback, 

•  Practice  based  on  mentoring  or  expert  instructional  guidance. 

This  was  termed  “practice  with  zeal”  by  Edwin  Thorndike  (1912),  one  of  the  founders  of 
educational  psychology  who  focused  on  classroom  learning  and  the  learning  of  simple  tasks. 
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More  recently,  it  has  been  termed  "deliberate  practice"  by  Ericsson  (e.g.,  Ericsson,  et  al.,  2006) 
with  reference  to  such  skills  as  in  music  and  chess.  But  the  notion  also  holds  for  such  domains  as 
science,  weather  forecasting,  engineering,  piloting,  and  military  command  (Hoffman,  2007).  The 
journeyman  is  one  who  has  practices  until  he  can  reliably  get  it  right.  Achievement  at  that  level 
can  be  accomplished  by  repetition.  The  expert  practices  until  he  can’t  get  it  wrong.  Mere 
repetition  is  not  sufficient  for  that.  What  is  necessary  is  practice  that  systematically  engages  the 
learner  in  increasingly  challenging  ways. 

The  key  features  of  High  Proficiency  are  listed  in  Table  2.1. 
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Table  2.1  Some  features  of  high  proficiency. 

•  The  expert  is  highly  regarded  by  peers. 

•  The  expert’s  judgments  are  accurate  and  reliable. 

•  The  expert’s  performance  shows  consummate  skill  and  economy  of  effort. 

•  For  routine  activities,  experts  display  “automaticity”  in  which  conscious  processing  is 
bypassed  and  the  expert  seems  to  be  carrying  out  a  task  without  significant  cognitive 
load. 

•  The  expert  possesses  knowledge  that  is  fine-grained,  detailed,  and  highly-organized. 

•  The  expert  knows  that  his  knowledge  is  constantly  changing  and  continually  contingent. 

•  The  expert  forms  rich  mental  models  of  cases  or  situations,  to  support  sensemaking  and 
anticipatory  thinking. 

•  The  expert  is  able  to  create  new  procedures  and  conceptual  distinctions. 

•  The  expert  is  able  to  cope  with  rare  and  tough  cases. 

•  The  expert  is  able  to  effectively  manage  resources  under  conditions  of  high  stakes  and 
high  risk  and  high  stress. 

•  Typically,  experts  have  special  knowledge  or  abilities  derived  from  extensive  experience 
with  sub-domains. 

•  The  expert  has  refined  pattern  perception  skills  and  can  apprehend  meaningful 
relationships  that  non-experts  cannot  apprehend 


When  an  expert  approaches  a  familiar  problematic  situation,  their  responses  do  not  tend  to  be 
from  an  analytical  or  deliberative  process  as  is  the  case  for  non-experts  (Frensch,  1988).  Rather, 
an  organized  set  of  memories  drawn  from  extensive  experience  forms  schemas  or  mental  models 
(Centner  &  Stevens,  1983),  which  give  meaning  and  structure  to  familiar  repeatedly  encountered 
problem  sets.  These  schemas  provide  intuitive,  immediate  cognitive  frameworks  to  help 
understand  the  nature  of  the  problem,  derive  potential  solutions,  and  anticipate  constraints 
(Reimann  &  Chi,  1989).  In  Recognition-Primed  Decision  Making,  the  recognition  of  case 
typicality  provides  intuitive,  immediate  sensemaking  frameworks  to  help  understand  the  nature 
of  the  problem,  derive  potential  solutions,  and  anticipate  constraints  (Klein,  1989;  Oliver  & 
Roos,  2005).  In  addition,  experts  are  able  to  recognize  aspects  of  a  problem  that  make  it  novel  or 
unusual,  and  will  bring  special  strategies  to  bear  to  solve  "tough  cases." 

What  about  criteria  for  high  proficiency  for  military  domains?  It  is  generally  assumed  that 
expertise  is  valuable  for  the  military,  but  how  can  we  know?  Casual  observation  suggests  that  the 
factors  noted  in  the  academic  literature  that  characterize  expertise  are  also  observed  in  highly 
accomplished  members  of  the  armed  services:  Self-assessment  and  self-directed  learning, 
deliberate  practice,  self-awareness  of  the  development  toward  expertise,  and  so  forth  (Fletcher, 
2009).  Success  at  conducting  military  operations  is  the  ultimate  criterion  against  which  these 
components  have  to  be  assessed.  As  is  true  for  expertise  in  general,  greater  intelligence  is  related 
to  better  performance  on  military  tasks,  but  high  intelligence  is  not  a  prerequisite  for  the 
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achievement  of  proficiency  of  expertise  (Ceci  &  Liker,  1986;  Lafrance,  1989).  For  military  jobs, 
as  is  true  for  expertise  in  general,  there  is  mixed  evidence  that  increased  experience  alone  is 
related  to  improved  performance  (Fletcher,  2009).  Rather,  success  seems  to  be  related  to  such 
additional  factors  as  cognitive  flexibility,  team  composition,  and  practice,  especially  deliberate 
practice. 

2.2.2  Proficiency  Scaiing  Concepts  and  Methods 

The  analysis  of  proficiency  and  proficiency  scaling  can  usefully  commence  by  distinguishing 
experts  (high  and  very  high  proficiency)  from  novices  (very  low  proficiency).  In  the 
psychological  and  educational  research  literatures,  expertise  has  defined  by  the  contrast  with 
novices.  Research  on  expertise  is  usually  premised  on  experimental  designs  have  these  two  basic 
proficiency  groups  as  the  main  comparison  (experts  vs.  novices  as  the  main  independent 
variable)  (cf.  Lajoie,  2003).  However,  there  is  considerable  variation  in  tacit  definitions  of 
“novice.”  For  example,  in  a  study  of  expert-novice  differences,  Chi,  Hutchinson,  and  Robin 
(1988)  relied  on  the  participation  of  an  avid  dinosaur  fan  who  happened  to  be  a  young  child.  In  a 
similar  developmental  study.  Means  and  Voss  (1985)  relied  on  the  participation  of  preschool 
children  who  were  avid  fans  of  the  "Star  Wars"  films.  In  many  academic  studies,  college 
students  have  served  as  experts  because  of  their  knowledge  of  particular  domains  (e.g.,  football, 
wedding  apparel,  regional  geography).  In  some  studies  of  expertise  in  the  solving  of  mechanics 
problems,  graduate  students  have  been  the  experts.  In  research  on  expertise  in  computer 
programming,  an  "expert"  can  be  someone  who  has  taken  a  few  courses,  and  would  therefore 
properly  be  called  an  apprentice)  (Mayer,  1988). 

Technically,  a  novice  is  a  person  who  has  begun  an  introductory  course  of  instruction.  Unlike  a 
“naive,”  who  is  a  person  who  may  not  even  know  that  the  domain  exists,  the  novice  is  the 
individual  who  has  committed  to  a  course  of  training. 

Many  studies  have  revealed  various  strategies  and  heuristics  that  experts  create  and  rely  upon 
(Hoffman,  2007;  Lajoie,  2003).  This  clearly  distinguishes  higher  from  lower  levels  of 
proficiency.  When  presented  with  domain  problems,  novices  and  apprentices  tend  to  rely  on 
superficial  features  and  rote  procedures  whereas  more  proficient  individuals  apprehend  the 
underlying  relational  structure,  the  constraints,  and  the  pertinent  principles  (Carlson,  1990;  Chi, 
Glaser,  &  Rees,  1982;  Egan  &  Schwartz,  1973;  Larkin,  1981;  Reed,  Dempster,  &  Ettinger, 
1985).  Experts  engage  in  rapid  apprehension  of  cases  and  paths  to  solutions,  whereas  novices 
engage  deliberate  processing  (Erensch,  1988). 

The  assumption  of  proficiency  scaling  is  that  across  the  developmental  continuum  there  are 
qualitative  shifts  (Dreyfus  &  Dreyfus,  1986;  Lajoie,  2003).  "The  knowledge  of  novices  is  not 
simply  an  incomplete  version  of  the  knowledge  of  the  expert  or  mature  learner,  it  is  qualitatively 
entirely  different"  (Cazden,  1976,  p.320). 

The  literature  of  Expertise  Studies  includes  many  discussions  of  the  notion  of  "automaticity" 
(James,  1890),  which  is  the  idea  that  knowledge  or  skill  undergoes  a  "declarative-to-procedural 
shift"  or  becomes  "routinized"  (Anderson,  1987).  In  a  number  of  cognitive  theories, 
developmental  levels  are  defined  entirely  in  terms  of  the  development  of  automaticity  and  a 
reliance  on  implicit  knowledge,  that  is,  procedural  knowledge  and  perceptual  skills  rather  than 
explicit  or  declarative  knowledge  (e.g.,  Benner,  1984;  Chamess  &  Campbell,  1988;  Dreyfus  & 
Dreyfus,  1986;  Eitts,  1964;  Eitts  &  Posner,  1967;  Gordon,  in  press;  Norman,  1987;  Rasmussen, 
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1986).  Dreyfus  and  Dreyfus  (1986;  Dreyfus,  1989)  proposed  a  five-level  theory  that  focuses 
exclusively  on  the  development  of  "intuition"  (i.e.,  non-verbalizable  procedural  knowledge  and 
metaknowledge),  which  they  regard  as  the  essence  of  proficiency. 

Most  such  theories  have  levels  such  as  those  presented  in  Table  2.2. 


Table  2.2.  Levels  of  proficiency  (after  Dreyfus  &  Dreyfus.  1986). 

Level  One.  Practitioners  at  this  level  have  knowledge  that  is  declarative  or  propositional,  their 
reasoning  is  said  to  be  explicit  and  deliberative.  Problem  solving  focuses  on  the  learning  of  facts, 
deliberative  reasoning,  and  a  reliance  on  general  strategies. 

Level  Two.  The  declarative  knowledge  of  practitioners  at  this  level  has  become  procedural  and 
domain- specific.  They  can  automatically  recognize  some  problem  types  or  situations. 

Level  Three.  At  this  level,  procedures  become  highly  routinized. 

Level  Four.  These  practitioners  are  proficient  and  have  a  great  deal  of  intuitive  skill. 

Level  Five.  Practitioners  at  this  highest  level  can  deliberately  reason  about  their  own  intuitions 
and  generate  new  rules  or  strategies  (what  Dreyfus  and  Dreyfus  call  "deliberative  rationality"). 


The  essence  of  all  such  theories  is  the  notion  that  reasoning  or  knowledge  originates  as  an 
analytic,  conscious,  deliberative,  step-wise  process,  and  evolves  into  rapid,  automatic,  non- 
conscious,  understanding  or  immediate  perceptual  judgments.  Many  researchers  in  psychology 
and  education  have  noted  what  seems  to  be  a  natural  developmental  sequence  from  superficial 
understanding,  to  a  semantic  understanding,  and  finally  to  a  qualitative  understanding  (Van 
Lehn,  2009). 

We  rely  on  a  modern  variation  on  the  classification  scheme  used  in  the  craft  guilds  (Renard, 
1968),  presented  in  Table  2.3. 
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Table  2.3.  Basic  proficiency  categories  based  on  the  traditional  craft  guild  terminology  (adapted 
from  Hoffman.  1998). 


Naive 

One  who  is  ignorant  of  a  domain. 

Novice 

Literally,  someone  who  is  new — a  probationary  member.  There  has  been  some 
(“minimal”)  exposure  to  the  domain. 

Initiate 

Literally,  someone  who  has  been  through  an  initiation  ceremony — a  novice 
who  has  begun  introductory  instruction. 

Apprentice 

Literally,  one  who  is  learning — a  student  undergoing  a  program  of  instruction 
beyond  the  introductory  level.  Traditionally,  the  apprentice  is  immersed  in  the 
domain  by  living  with  and  assisting  someone  at  a  higher  level.  The  length  of  an 
apprenticeship  depends  on  the  domain,  ranging  from  about  one  to  12  years  in 
the  craft  guilds. 

Journeyman 

Literally,  a  person  who  can  perform  a  day’s  labor  unsupervised,  although 
working  under  orders.  An  experienced  and  reliable  worker,  or  one  who  has 
achieved  a  level  of  competence.  It  is  possible  to  remain  at  this  level  for  life. 

Expert 

The  distinguished  or  brilliant  journeyman,  highly  regarded  by  peers,  whose 
judgments  are  uncommonly  accurate  and  reliable,  whose  performance  shows 
consummate  skill  and  economy  of  effort,  and  who  can  deal  effectively  with 
certain  types  of  rare  or  “tough”  cases.  Also,  an  expert  is  one  who  has  special 
skills  or  knowledge  derived  from  extensive  experience  with  subdomains. 

Master 

Traditionally,  a  master  is  any  journeyman  or  expert  who  is  also  qualified  to 
teach  those  at  a  lower  level.  Traditionally,  a  master  is  one  of  an  elite  group  of 
experts  whose  judgments  set  the  regulations,  standards,  or  ideals.  Also,  a  master 
can  be  that  expert  who  is  regarded  by  the  other  experts  as  being  “the”  expert,  or 
the  “real”  expert,  especially  with  regard  to  sub-domain  knowledge. 

Proficiency  scaling  is  the  attempt  to  forge  a  domain-  and  organizationally-appropriate  scale  for 
distinguishing  levels  of  proficiency.  Some  alternative  methods  are  described  in  Table  2.4. 
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Table  2.4.  Some  methods  that  can  contribute  data  for  the  creation  of  a  proficiency  scale. 


Method 

Yield 

Example 

In-depth  career 
interviews 
about 
education, 
training,  etc. 

Ideas  about  breadth 
and  depth  of 
experience; 

Estimate  of  hours  of 
experience 

Weather  forecasting  in  the  armed  services,  for  instance, 
involves  duty  assignments  having  regular  hours  and 
regular  job  or  task  assignments  that  can  be  tracked 
across  entire  careers.  Amount  of  time  spent  at  actual 
forecasting  or  forecasting-related  tasks  can  be 
estimated  with  some  confidence  (Hoffman,  1991). 

Professional 
standards  or 
licensing 

Ideas  about  what  it 
takes  for  individuals 
to  reach  the  top  of 
their  field. 

The  study  of  weather  forecasters  involved  senior 
meteorologists  US  National  Atmospheric  and 
Oceanographic  Administration  and  the  National 

Weather  Service  (Hoffman,  Coffey,  &  Ford,  2000). 

One  participant  was  one  of  the  forecasters  for  Space 
Shuttle  launches;  another  was  one  of  the  designers  of 
the  first  meteorological  satellites. 

Measures  of 
performance  at 
the  familiar 
tasks 

Can  be  used  for 
convergence  on 
scales  determined  by 
other  methods. 

Weather  forecasting  is  again  a  case  in  point  since 
records  can  show  for  each  forecaster  the  relation 
between  their  forecasts  and  the  actual  weather.  In  fact, 
this  is  routinely  tracked  in  forecasting  offices  by  the 
measurement  of  "forecast  skill  scores"  (see  Hoffman  & 
Trafton,  forthcoming). 

Social 

Interaction 

Analysis 

Proficiency  levels  in 
some  group  of 
practitioners  or 
within  some 
community  of 
practice  (Mieg, 

2000;  Stein,  1997) 

In  a  project  on  knowledge  preservation  for  the  electric 
power  utilities  (Hoffman  &  Hanes,  2003),  experts  at 
particular  jobs  (e.g.,  maintenance  and  repair  of  large 
turbines,  monitoring  and  control  of  nuclear  chemical 
reactions,  etc.)  were  readily  identified  by  plant 
managers,  trainers,  and  engineers.  The  individuals 
identified  as  experts  had  been  performing  their  jobs  for 
years  and  were  known  among  company  personnel  as 
"the"  person  in  their  specialization:  "If  there  was  that 
kind  of  problem  I'd  go  to  Ted.  He's  the  turbine  guy. " 

A  proficiency  scale  for  a  given  domain  should  be  based  on  more  than  one  of  the  four  general 
types  of  method  listed  in  Table  2.4,  and  ideally  should  be  based  on  at  least  three.  This  has  been 
referred  to  as  the  “three  legs  of  a  tripod"  (Hoffman,  Ford  &  Coffey,  2000;  Hoffman  &  Lintern, 
2006). 

Social  Interaction  Analysis,  the  result  of  which  is  a  sociogram,  is  perhaps  the  lesser  known  of  the 
four  kinds  of  methods.  A  sociogram,  that  represents  interaction  patterns  between  people  (e.g., 
frequent  interactions),  is  used  to  study  group  clustering,  communication  patterns,  and  workflows 
and  processes.  For  Social  Interaction  Analysis,  multiple  individuals  within  an  organization  or  a 
community  of  practice  are  interviewed.  Practitioners  might  be  asked,  for  example,  "If  you  have  a 
problem  of  type  x,  who  would  you  go  to  for  advice?"  or  they  might  be  asked  to  sort  cards 
bearing  the  names  of  other  domain  practitioners  into  piles  according  to  one  or  another  skill 
dimension  or  knowledge  category. 
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It  is  important  to  create  a  scale  that  is  both  domain  and  organizationally-appropriate,  and  that 
considers  the  full  range  of  proficiency.  In  the  project  on  weather  forecasting  (Hoffman,  Coffey  & 
Ford,  2000),  the  proficiency  scale  distinguished  three  levels:  experts,  journeymen,  and 
apprentices,  but  each  of  these  was  further  distinguished  by  three  levels  of  seniority,  up  to  the 
senior  experts  who  has  as  much  as  50,000  hours  of  practice  and  experience  at  domain  tasks. 

Although  mere  years  of  experience  does  not  guarantee  expertise,  there  is  a  partial  linkage  in  that 
the  longer  the  career  the  greater  the  opportunity  for  more  experience,  and  more  diverse 
experiences.  Experience  scaling  can  also  involve  an  attempt  to  gauge  the  practitioners'  breadth  of 
experience — the  different  contexts  or  sub-domains  they  have  worked  in,  the  range  of  tasks  they 
have  conducted,  etc.  Deeper  experience  (i.e.,  more  years  at  the  job)  affords  more  opportunities  to 
acquire  broader  experience.  Hence,  depth  and  breadth  of  experience  should  be  correlated.  But 
they  are  not  necessarily  correlated,  and  so  examination  of  both  breadth  and  depth  of  experience 
is  always  wise.  Thus,  one  comes  to  a  resolution  that  age  and  proficiency  are  generally  related, 
that  is,  partially  correlated.  This  is  captured  in  Table  2.5. 
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Table  2.5.  Age  and  proficiency  are  partially  correlated. 


Lieespan 

10 

20  30  40  50  60 

Novice 

Learning  can  commence  at  any  age 
(e.g.,  the  school  child  who  is  avid  about 
dinosaurs;  the  adult  professional  who 
undergoes  job  re-training). 

Achievement  of  expertise  in 
significant  domains  may  not 
be  possible 

Intern  & 
Apprentice 

Individuals 
less  thanlSyrs 
of  age  are 
rarely 
considered 
appropriate  as 
Apprentices 

Can  commence  at  any  age  ^  ^ 

Achievement  of 
expertise  in 
significant 
domains  may 
not  be  possible 

Journeyman 


It  is  possible 
to  achieve 
journeyman 
status  (e.g., 
chess, 
computer 
hacking, 
sports) _ 


Is  typically  achieved  in  mid-  to  late-20s,  but 
development  may  go  no  further 


Expert 

Stage 


It  is  possible  to 
achieve  expertise 
(e.g.,  chess, 
computer  hacking, 
sports) 


Most  typical  for  35  yrs  of  age  and  beyond 


Master 

Stage 

Is  rarely 
achieved  early 
in  a  career 

It  is  possible  to 
achieve  mid¬ 
career 

Most  typical  of  seniors 

In  the  proficiency  scaling  in  weather  forecasting,  Hoffman,  Coffey  and  Ford  (2000)  conducted 
interviews,  sociometric  analysis  and  performance  evaluation  at  a  U.S.  Navy  weather  forecasting 
facility  situated  at  an  airfield.  By  comparison  of  the  forecasts  with  forecast  verification  data,  it 
was  possible  to  scale  performance.  By  detailed  analysis  of  personnel  records  (duty  and  duty 
assignments),  it  was  possible  to  estimate  hours  spent  at  forecasting-related  activity.  Through 
sociometry  and  career  interviews  it  was  possible  to  gauge  depth  and  breadth  of  experience.  Depth 
of  experience  (number  of  hours)  was  compared  to  breadth  of  experience,  as  indicated  by  the 
variety  of  experiences  possible — forecasting  while  at  sea,  experience  in  more  than  one  climate, 
and  so  on.  The  expectation  was  that  depth  of  experience  (number  of  hours)  would  be  verified  by 


37 


Report  on  FA8650-09-2-6033 


breadth  (variety)  of  experience,  if  only  because  longer  experience  affords  more  opportunities  for  a 
greater  variety  of  experiences.  By  using  these  combined  methods,  the  researchers  converged  on  a 
proficiency  scale  that  included  levels  (expert,  journeyman,  apprentice)  and  sub-levels  (e.g.,  junior, 
senior).  Table  2.6  describes  the  skill  factors  definitive  of  each  level  in  the  scale. 


Table  2.6.  Skill  factors  forming  a  proficiency  scale  appropriate  to  U.  S.  Navy  forecasting. 


Apprentice 

JOURNEYMAN 

Expert 

Senior  Expert 

Process 

Forecasting  by 
extrapolation  from 
the  previous  weather 
and  forecast  and  by 
reliance  on 
computer  models. 

Begins  by 
formulating  the 
problem  of  the  day 
but  focuses  on 
forecasting  by 
extrapolation  from 
the  previous  weather 
and  forecast  and  by 
reliance  on 
computer  models. 

Begins  by 
formulating  the 
problem  of  the  day 
and  then  building  a 
mental  model  to 
guide  further 
information  search 

Begins  by 
formulating  the 
problem  of  the  day 
and  then  building  a 
mental  model  to 
guide  further 
information  search 

Strategy 

Reasoning  is  at  the 
level  of  individual 
cues  within  data 
types 

Reasoning  is  mostly 
at  the  level  of 
individual  cues, 
some  ability  to 
recognize  cue 
configurations 
within  and  across 
data  types 

Reasoning  is  in 
terms  of  both  cues 
and  cue 

configurations,  both 
within  and  across 
data  types.  Some 
recognition-primed 
decision-making 
occurs 

Process  of  mental 
model  formation  and 
refinement  is  more 
likely  to  be  short- 
circuited  by 
recognition-primed 
decision-making 
skill 

For  purposes  of  this  Report  we  must  repeatedly  regain  focus  on  a  proficiency  scale  beginning  at 
the  apprentice  level  and  including  the  intermediate  journeyman  category,  and  a  scale  in  which 
the  category  of  “novice”  as  — a  person  who  knows  next  to  nothing  about  a  domain — is  not  nearly 
as  pertinent  as  the  concept  of  the  junior  journeyman.  In  a  review  of  the  literature  of  expertise 
studies  written  in  1999,  Hoffman  et  al.  said: 

Relatively  little  research  has  looked  at  the  midrange — the 
distinctive  qualities  of  apprentice-level  and  journeyman-level 
individuals  (Bereiter  &  Bird,  1985;  Campbell,  Brown,  &  DiBello, 

1992;  Glaser,  1976;  Mayer,  1988).  Some  exceptions  are  Means  and 
Voss's  (1985)  study  of  the  development  of  knowledge  about  the 
"Star  Wars"  films,  with  participants  spanning  ages  7  to  19  years, 
and  Myles-Worsley,  Johnston,  &  Simon's  (1988)  study  of  initiate 
(student)  and  journeyman  (intern),  and  well  as  expert  (resident) 
radiologists  (Ch.  3.) 
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This  statement  is  still  true. 


2.2.3  What  makes  the  Achievement  of  High  Proficiency  Difficuit? 

Studies  that  have  attempted  to  create  domain- appropriate  proficiency  scales  (e.g.,  Ericsson,  et  ah, 
1993;  Hoffman,  et  ah,  2002)  have  shown  that  the  amount  of  practice  is  the  single  greatest 
determinate  of  superior  expertise,  followed  by  specificity  of  practice  and  deliberateness  of 
practice  to  task  performance  (Schmidt  &  Lee,  2005;  Ericsson,  2003).  Expertise  is  achieved  only 
after  working  hard,  on  hard  problems  (Eccles,  et  ah,  2009).  Eor  instance,  in  the  field  of 
continuing  education  it  is  recognized  that  the  progression  to  proficiency  depends  on  practice  at 
difficult  tasks.  Valkeavaara  (1999)  interviewed  a  group  of  highly  experienced  workers  in  the 
field  of  human  resource  management  to  reveal  the  features  of  problems  that  made  them  difficult. 
The  semi- structured  interviews  confirmed  that  the  best  "lessons  learned"  were  in  situations  that 
placed  new  demands  on  the  worker. 

Research  by  Paul  Eeltovich,  Rand  Spiro  (Working  Meeting  Participants)  and  their  colleagues  on 
expert-novice  differences  in  domains  of  medicine  investigated  how  people  deal  with  difficult, 
challenging  cases  (e.g.,  Eeltovich,  Spiro,  &  Coulson,  1989;  Eeltovich,  Coulson,  &  Spiro,  2001). 
That  research  identified  characteristics  of  problems  that  cause  cognitive  difficulty  for  learners, 
but  not  for  experts.  It  also  revealed  how  people  respond  to  these  elements  of  difficulty  in 
learning  and  in  the  application  of  knowledge. 

The  research  was  initially  driven  by  the  identification  of  important  biomedical  science  concepts 
that  routinely  cause  difficulty  among  students  for  learning,  understanding  and  application.  These 
were  identified  by  large-scale  polling  of  experts  (medical  school  teachers)  across  the  United 
States  and  Canada.  They  were  asked  to  propose  areas  of  basic  biomedical  science  that  were  both 
important  to  the  practice  of  medicine  and  also  very  difficult  for  learners  to  master  (Dawson- 
Saunders,  Eeltovich,  Coulson,  &  Steward,  1990;  see  also  Patel,  Kaufman,  &  Magder,  1991).  The 
resulting  compilation  was  used  to  guide  the  research  on  cognitive  responses  to  difficulty,  using 
cognitive  task  analysis  methods  such  as  think-aloud  problem  solving.  The  research  identified 
characteristics  of  subject  matter  or  situations  that  cause  difficulty  for  learners,  including  learners 
nearing  the  journeyman  level  of  proficiency  (Eeltovich,  Spiro,  &  Coulson,  1989;  Eeltovich, 
Spiro,  &  Coulson,  1993;  Eeltovich,  Coulson  &  Spiro,  2001).  The  “dimensions  of  difficulty”  that 
make  tasks  difficult  and  require  mental  effort,  on  the  part  of  learners,  but  that  are  less 
troublesome  for  experts,  are  presented  in  Table  2.7. 
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Table  2.1 .  The  Dimensions  of  Difficulty. 


Static  versus  dynamic 

Are  important  aspects  of  a  situation  captured  by  a  fixed  “snapshot,”  or  are  the  critical 
characteristics  captured  only  by  the  changes  from  frame  to  frame?  Are  phenomena  static  and 
scalar,  or  do  they  possess  dynamic  vector  characteristics? 

Discrete  versus  continuous 

Do  processes  proceed  in  discernable  steps,  or  are  they  unbreakable  continua?  Are  attributes 
describable  by  a  small  number  of  categories  (e.g.,  dichotomous  classifications  like  large/small), 
or  is  it  necessary  to  recognize  and  utilize  entire  continuous  dimensions  (e.g.,  the  full  dimension 
of  size)  or  large  numbers  of  categorical  distinctions? 

Separable  versus  interactive 

Do  processes  occur  independently  or  with  only  weak  interaction,  or  is  there  strong  interaction 
and  interdependence? 

Sequential  versus  simultaneous 

Do  processes  occur  one  at  a  time,  or  do  multiple  processes  occur  at  the  same  time? 

Homogeneous  versus  heterogeneous 

Are  components  or  explanatory  schemes  uniform  (or  similar)  across  a  system — or  are  they 
diverse? 

Single  versus  multiple  representations 

Do  elements  in  a  situation  afford  single  (or  just  a  few)  interpretations,  functional  uses, 
categorizations,  and  so  on,  or  do  they  afford  many?  Are  multiple  representations  (e.g.,  multiple 
perspectives,  schemas,  analogies,  case  precedents,  etc.)  required  to  capture  and  convey  the 
meaning  of  a  process  or  situation? 

Mechanism  versus  organicism 

Are  effects  traceable  to  simple  and  direct  causal  agents,  or  are  they  the  product  of  more  system- 
wide,  organic  functions?  Can  important  and  accurate  understandings  be  gained  by 
understanding  just  parts  of  the  system,  or  must  the  entire  system  be  understood  for  even  the 
parts  to  be  understood  well? 

Linear  versus  nonlinear 

Are  functional  relationships  linear  or  nonlinear  (i.e.,  are  relationships  between  input  and  output 
variables  proportional  or  non-proportional)?  Can  a  single  line  of  explanation  convey  a  concept 
or  account  for  a  phenomenon,  or  are  multiple  overlapping  lines  of  explanation  required  for 
adequate  coverage? 

Table  continues 
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Table  continued 


Universal  versus  conditional 

Do  guidelines  and  principles  hold  in  much  the  same  way  (without  the  need  for  substantial 
modification)  across  different  situations,  or  is  there  great  context-sensitivity  in  their 
application? 

Regular  versus  irregular 

Is  a  domain  characterized  by  a  high  degree  of  routinizabilty  or  prototypicality  across  cases,  or 
do  cases  differ  considerably  from  each  other  even  when  they  are  called  by  the  same  name?  Are 
there  strong  elements  of  symmetry  and  repeatable  patterns  in  concepts  and  phenomena,  or  is 
there  a  prevalence  of  asymmetry  and  absence  of  consistent  pattern? 

"Surface”  versus  "deep” 

Are  important  elements  for  understanding  and  for  guiding  action  delineated  and  apparent  on  the 
surface  of  a  situation,  or  are  they  more  covert,  relational,  abstract? 


When  the  material  to  be  learned  or  understood  is  characterized  by  the  second  alternative  in  each 
of  the  pairs  in  Table  2.2,  then  learners  will  tend  to  interpret  the  situations  as  though  they  were 
characterized  by  the  simpler  alternatives,  their  understandings  will  tend  to  be  reductive,  that  is, 
they  will  tend  to  simplify,  they  will  tend  to  attempt  to  defend  their  simple  understandings  when 
confronted  with  facts  that  suggest  that  the  situation  is  more  complex  than  what  they  suppose. 
These  "knowledge  shields"  will  allow  them  to  maintain  their  incorrect  beliefs  and 
understandings.  It  requires  effort,  practice,  experience  (i.e.,  the  achievement  of  expertise)  for 
people  to  begin  to  overcome  the  knowledge  shields. 

This  is  the  “reductive  tendency,”  the  inclination  for  learners  to  construct  understandings  and 
categories  that  are  overly  simplistic.  The  reductive  tendency  is  an  inevitable  consequence  of  how 
people  learn  (Feldman,  2003;  Feltovich,  Hoffman  &  Woods,  2004).  That  is,  when  people  are 
forming  a  new  understanding  or  developing  a  new  category  their  knowledge  is  necessarily 
incomplete.  It  is  only  through  additional  experience  and  thought  that  distinctions  come  to  be 
perceived,  understood,  and  learned.  So  at  any  point  in  time,  the  human's  understanding  of 
anything  that  is  at  all  complex,  even  some  of  the  understandings  held  by  domain  experts,  is 
bound  to  be  simplifying  at  least  in  some  respects.  In  the  areas  of  complex  cognition  studied  (e.g., 
the  learning  of  clinical  medicine),  the  reductive  tendency  has  been  shown  to  lead  to  significant 
misconceptions  (e.g.,  Coulson,  Feltovich,  &  Spiro,  1989;  Feltovich,  Coulson  &  Spiro,  2001; 
Spiro,  Feltovich,  Coulson,  &  Anderson,  1989),  and  in  some  instances  to  degraded  medical  care 
(Coulson,  Feltovich,  &  Spiro,  1997).  In  addition,  the  misconceptions  and  errors  that  develop  can 
be  resistant  to  change,  an  obstacle  to  the  achievement  of  expertise.  When  learners  are  confronted 
with  evidence  contrary  to  their  views,  they  engage  mental  maneuvers  to  rationalize  their  faulty 
beliefs  without  fundamentally  altering  their  views.  These  are  the  “knowledge  shields.”  Twenty- 
three  of  them  have  been  identified  (Feltovich  et  ah,  2001). 

Although  this  research  first  had  its  impact  primarily  in  the  domain  of  biomedical  science  as  it 
relates  to  clinical  medicine,  it  has  found  widespread  application.  Diverse  areas  where  the 
research  on  the  dimensions  of  difficulty  and  the  reductive  tendency  have  seen  application  and 
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validation  include  biology  (e.g.,  Patel,  Kaufman,  &  Magder,  1991),  complex  systems  (e.g., 
Hmelo-Silver  &  Pfeffer,  in  press;  Resnick,  1996),  lay-people's  understanding  of  disease  (Furze, 
Roebuck,  Bull,  Lewin,  &  Thompson,  2002),  work-place  error  and  mishap  (Cook  &  Woods, 
1994),  military  command  and  control  (Houghton,  Leedom  &  Miles,  2002),  and  the  process  of 
design,  especially  the  design  of  complex  sociotechnical  systems  (e.g..  Woods,  2002). 

Studies  of  experts  and  expert-novice  differences  also  suggested  that  the  achievement  of 
proficiency  depends  on  the  learner’s  active,  deliberative  creation  and  refinement  of  their  own 
knowledge.  The  Cognitive  Flexibility  Theory  of  Paul  Feltovich,  Rand  Spiro,  and  their  colleagues 
(Spiro,  Feltovich,  Jacobson  &  Coulson,  1991,  1992;  Spiro  &  Jheng,  1990)  asserts  that  people 
(especially  experts)  have  the  ability  to  restructure  their  knowledge  to  adapt  to  complex,  ill- 
structured  problems.  Cognitive  flexibility  is  “the  ability  to  represent  knowledge  from  different 
conceptual  and  case  perspectives  and  then,  when  the  knowledge  must  later  be  used,  the  ability  to 
construct  from  those  different  conceptual  and  case  representations  a  knowledge  ensemble 
tailored  to  the  needs  of  the  understanding  or  problem-solving  situation  at  hand"  (Spiro,  et  ah, 
1992,  p.  58).  This  ability  hinges  on  experience  with  diverse  problem  types,  and  opportunities  to 
understand  the  material  from  differing  perspectives.  This  enables  people  to  refine  their 
understanding  of  concepts  and  the  relations  among  concepts,  in  processes  such  as  those 
discussed  by  Jean  Piaget  (e.g.,  Inhelder  &  Piaget,  1958)  and  David  Ausubel  (1968;  Ausubel, 
Novak,  &  Hanesian,  1978)  such  as  assimilation  and  accommodation.  The  theory  also  asserts  that 
knowledge  and  knowledge-in-use  are  context-dependent,  and  so  instruction  needs  to  be  with 
reference  to  meaningful  contexts.  This  entails  some  straight-forward  guidelines  for  effective 
instruction: 

•  Learning  activities  must  provide  multiple  representations  of  content. 

•  Instructional  materials  should  avoid  oversimplifying  the  content  domain  and  support 
context  dependent  knowledge. 

•  Instruction  should  be  case-based  and  emphasize  knowledge  construction,  not 
transmission  of  information. 

•  Knowledge  sources  should  be  highly  interconnected  rather  than  compartmentalized. 

These  guidelines  have  been  widely  and  successfully  applied  (e.g.,  Jonassen,  Ambruso,  &  Olesen, 
1992). 

2.3  Training  and  Practice 

Most  traditional  training  covers  what  is  generally  familiar  and  well-known  to  trainers,  educators 
and  training  establishments  in  the  context  of  bounded  operational  and  organizational  settings.  As 
such  the  ‘training  for  the  test’  measures  can  be  base-lined  and  validated  to  create  normative 
performance  scores.  It  is  still  appropriate  for  training  in  many  jobs/tasks  to  rely  on  checklist 
evaluation  based  on  correct  or  expected  answers  or  fixed  procedures. 

However,  the  need  for  or  a  capacity  for  critical  thinking  (in  terms  of  thinking  laterally  and 
innovatively  or  being  able  to  improvise)  and  capability  to  engage  with  what  cannot  be  or  has  not 
been  foreseen  is  central  to  many  existing  and  emerging  jobs/tasks,  including  many  emerging 
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military  jobs  (e.g.,  cultural  analysis,  SASO  operations,  PIMESII,  etc.)  (Center  for  Applied 
Systems  Studies,  2010). 

If  there  can  be  any  one  generalization  about  the  science  of  training  and  instructional  design  it  is: 

Generalization  is  difficult. 

2.3. 1  Types  of  Practice  and  Training  Sequences 

Massed  versus  Distributed  Practice 

A  long  line  of  psychological  research  has  shown  the  beneficial  effect  of  spacing  on  learning  and 
memory  when  the  goal  is  long  retention  intervals  (e.g.,  Bahrick,  Bahrick,  Bahrick,  &  Bahrick, 
1993;  Bjork  &  Allen,  1970;  Bloom  &  Schuell,  1981;  Dempster,  1988;  Glenberg,  1976;  Glenberg 
&  Lehmann,  1980;  Greene,  1989;  Reynolds  &  Glaser,  1964;  Schmidt  &  Bjork,  1992). 

Some  studies  find  that  optimal  performance  at  test  seems  to  result  when  the  interval  between 
practice  trials  is  approximately  equal  in  length  to  the  retention  interval.  This  conclusion  seems  to 
hold  even  when  practice  is  distributed  across  sessions  rather  than  across  trials  within  a  session 
(Healy,  2007.) 

With  respect  to  military  tasks  looking  at  massed  versus  distributed  practice,  similar  evidence  has 
been  found.  For  example,  Hagman  (1980)  varied  this  type  of  practice  in  a  maintenance  task  and 
found  that,  when  tested  two  weeks  later  the  participants  in  the  massed  practice  condition 
committed  40%  more  errors  and  took  51%  longer  than  the  distributed  practice  group. 

This  finding  has  a  direct  implication  for  retraining  following  hiatus.  Many  ideas  about  skill 
retention  during  hiatus  involve  practice  experiences  that  would  inevitably  be  massed  in  the  sense 
of  concentrated  at  particular  times  within  the  practitioner’s  primary  work  schedule.  The  literature 
on  distributed  practice  would  suggest  that  training  during  hiatus  be  distributed. 

Blocked  versus  Scheduled  Practice 

Blocked  practice  refers  to  practice  in  which  there  are  successive  trials  that  repeat  a  particular 
activity.  This  is  contrasted  with  a  randomized  schedule  in  which  the  same  task  is  never  practiced 
on  successive  trials,  but  is  randomized  with  other  practices.  A  number  of  researchers  have  found 
that  retention  is  better  in  random  versus  blocked  practice  schedules  (Landauer  &  Bjork,  1978; 
Rea  &  Modigliani,  1985).  This  holds  true  for  both  simple  motor  skills  and  simple  verbal  memory 
tasks.  It  is  theorized  that  randomized  practice  forces  the  learner  to  continuously  engage  with  new 
and  difficult  information  to  process,  which  can  be  to  the  detriment  of  initial  retention,  but 
prevents  reliance  on  a  stable  set  of  “superficial  massed  rehearsal.”  This  allows  for  greater  long¬ 
term  retention. 

Sequencing  by  Content  Type 

There  is  considerable  evidence  that  training  experiences  need  to  be  sequenced  with  respect  to  the 
“knowledge  type.”  Koubek  and  Clarkston  (2004)  demonstrated  that  training  can  affect  the 
development  of  knowledge  structures  which  can  in  turn  affect  training  effectiveness.  Their  study 
focused  on  the  question  of  whether  trainees  benefit  more  from  first  learning  about  domain 
concepts  and  principles,  or  first  learning  about  specific  details  of  tasks,  and  procedures.  Students 
with  no  prior  knowledge  about  plastic  extrusion  machines  were  taught  about  plastic  extrusion  by 
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presentations  in  one  of  two  orders  involving  the  two  training  topics  of  domains  concepts  and 
principles,  or  by  versus  details  of  processes.  Trainee  knowledge  was  assessed  by  asking  for 
trainees  to  indicate  the  relations  between  all  possible  pairs  of  domain  concepts  (e.g.,  mold- 
warpage,  resting  period-barrel).  These  pair-wise  judgments  could  be  used  to  form  hierarchical 
models  of  the  trainees'  "knowledge  structures,"  which  could  be  evaluated  for  correctness, 
completeness,  and  complexity  (number  of  hierarchical  levels).  Overall,  the  group  receiving  the 
first  training  about  concepts  and  principles  did  better  on  the  final  test  (i.e.,  more  levels  in  the 
knowledge  hierarchies  and  more  interconnections  among  concepts). 

One  must  note  that  the  participants  were  naive  to  the  domain  at  the  outset,  and  also  received  no 
actual  experience  with  plastic  extrusion  machines.  Likely  the  ordering  of  the  conditions,  and  the 
resultant  effects,  would  differ  were  one  to  have  apprentices  or  journeymen  with  actual 
experience  as  the  participants. 

Sequencing  by  Complexity 

It  is  sometimes  assumed  that  the  way  to  help  learners  cope  with  complexity  is  to  simplify  the 
situation  and  then  incrementally  introduce  increasingly  complex  elements.  However,  initial 
learning  that  is  more  difficult  can  lead  to  greater  flexibility  and  transfer.  Research  on  high-end 
learning  (e.g.,  in  medical  education)  has  shown  that  when  learners  are  initially  exposed  to 
simplifications  of  complex  topics,  serious  misunderstandings  can  remain  entrenched  and 
interfere  with  or  preclude  deeper,  more  accurate  understandings  (Feltovich,  Spiro,  &  Coulson, 
1993).  Complex  tasks  are  typically  taught  as  a  procedure  when  it  is  more  appropriate  to  train  it 
as: 

“...a  combination  of  domain  and  system  knowledge  (conceptual 
models  of  the  system  including  system  components  and 
interaction,  flow  controls,  fault  states,  fault  characteristics, 
symptoms,  contextual  information  and  probabilities  of 
occurrence);  troubleshooting  strategies  such  as  search  and  replace, 
serial  elimination,  and  space  splitting;  and  fault  testing  procedures” 

(Jonassen,  2007,  p.  12). 

Sequencing  by  Readiness  Level 

A  well-known  finding  in  research  on  text  comprehension  is  that  low-knowledge  and  high- 
knowledge  readers  learn  different  things  and  in  different  ways  (e.g.,  MacNamara,  et  ah,  1996). 

Based  on  a  review  of  research  on  student  learning  from  textbooks  and  diagrams,  Mayer  (2001) 
concluded  that  inexperienced  learners  benefit  far  more  from  instructional  presentations  designed 
to  provide  a  better  support  for  cognitive  processes  than  high-knowledge  learners.  Less 
knowledgeable  learners  benefit  from  additional,  integrative  material  whereas  more 
knowledgeable  learners  do  not. 

Kalyuga,  et  al.  (2003)  reviewed  the  literature  showing  that  learning  depends  on  the  level  of 
proficiency  (expertise)  of  the  learner.  Instructional  techniques  that  are  highly  effective  with 
inexperienced  learners  can  lose  their  effectiveness  and  even  have  negative  consequences  when 
used  with  more  experienced  learners.  Conversely,  instructional  methods  that  work  for  more 
advanced  learners  may  be  ineffective  for  less  advanced  learners.  One  hypothesis  has  been  that 
"schemas"  stored  in  long-term  memory  (said  to  be  a  defining  feature  of  expert  memory 
organization)  reduce  working  memory  load  during  learning  and  thereby  aid  learning.  For  less 
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advanced  learners,  instructional  guidance  can  compensate  for  the  learner's  lack  of  schemas. 

If  the  instructional  presentation  fails  to  provide  necessary  guidance, 
learners  will  have  to  resort  to  problem-solving  search  strategies  that 
are  cognitively  inefficient  because  they  impose  a  heavy  working 
memory  load  (p.  24). 

In  contrast,  since  more  advanced  learners  bring  their  memory  schemas  to  bear,  if  the  problem 
task  includes  integrative  material,  this  can  be  a  distraction. 

This  additional  cognitive  load  may  be  imposed  even  if  a  learner 
recognizes  the  instructional  materials  to  be  redundant  and  so 
decides  to  ignore  that  information  as  best  he  or  she  can  (p.  24). 

For  the  novice  learner,  excessive  working  memory  load  inhibits  the  ability  to  adequately  process 
material  if  it  is  beyond  their  level  of  expertise.  For  experts,  on  the  other  hand,  learning  is 
inhibited  if  they  are  forced  to  work  with  material  below  their  level  of  expertise,  because  it  forces 
redundant  processing.  This  is  referred  to,  somewhat  misleadingly,  as  the  expertise  reversal 
effect:  experts  are  more  likely  learn  if  they  are  given  the  opportunity  to  engage  in  exploratory 
problem  solving  that  engages  previously  acquired  schemas,  whereas  novices  are  likely  to  benefit 
from  worked  examples  that  do  not  involve  exploratory  problem  solving.  This  accords  with  the 
guidelines  of  Problem-Based  Learning. 

In  a  study  of  comprehension  of  electrical  circuit  diagrams,  Kalyuga,  Chandler,  and  Sweller 
(1998)  found  that  less  experienced  electrical  trainees  benefitted  from  textual  explanations 
accompanying  the  circuit  diagrams  that  were  not  able  to  comprehend  in  a  diagram-only  format. 
More  experienced  trainees  performed  significantly  better  with  the  electrical  circuit  diagram-only 
format  and  reported  less  mental  effort  associated  with  studying  the  diagram-only  format. 

A  host  of  studies  have  shown  that  when  learning  to  solve  problems  of  a  given  kind,  examples 
help.  Beyond  this  finding,  presenting  example  problems  that  are  fully  worked-out  can  sometimes 
be  as  good  an  instructional  method  as  presenting  incomplete  problems  that  the  learner  has  to 
work  out  (e.g.,  Trafton  &  Reiser,  1993).  However,  this  too  can  depend  on  the  learner's 
proficiency  level.  Once  a  student  has  learned  to  solve  a  problem  type  to  a  reasonable  level  of 
proficiency,  when  problem  solving  becomes  less  effortful,  presenting  a  fully  worked  example 
that  the  student  has  to  work  through  can  entail  a  cognitive  load  greater  than  that  of  problem 
solving  (Kalyuga,  et  ah,  2001). 

...  inexperienced  learners  benefitted  most  from  an  instructional 
procedure  that  placed  a  heavy  emphasis  on  guidance.  Any 
additional  instructional  guidance  (e.g.,  indicating  a  goal  or 
subgoals  associated  with  a  task,  suggesting  a  strategy  to  use, 
providing  solution  examples,  etc.)  should  reduce  cognitive  load  for 
inexperienced  learners,  especially  in  the  case  of  structurally 
complex  instructional  materials.  At  the  same  time,  additional 
instructional  guidance  might  be  redundant  for  more  experienced 
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learners  and  require  additional  working  memory  resources  to 
integrate  the  instructional  guidance  with  learners’  available 
schemas  that  provide  essentially  the  same  guidance.  A  minimal 
guidance  format  might  be  more  beneficial  for  these  learners 
because  they  are  able  to  use  their  schema-based  knowledge  as 
guidance  in  constructing  integrated  mental  representations  without 
overloading  working  memory  (Kalyuga,  et  al.,  2003,  p.  27). 

Such  an  interaction  also  obtains  for  the  effect  of  mental  practice,  which  is  more  beneficial  for 
more  advanced  learners  than  less  advanced  learners.  This  is  also  explained  on  the  hypothesis  that 
advanced  learners  have  developed  mental  schemas  (Cooper,  et  al.,  2001). 

The  findings  and  effects  described  here  depend  on  the  nature  of  the  to-be-learned  material.  Some 
material  consists  of  elements  or  chunks  that  can  be  understood  separately,  whereas  other  material 
consists  of  elements  or  chunks  that  have  to  be  understood  as  an  interacting  whole.  If  a  learner 
understands  material  in  an  integrated  way,  they  will  have  formed  schemas  that  can  be  used  to 
organize  information  presented  in  new  problems.  Novices  need  to  be  presented  the  information 
in  such  a  way  that  they  can  understand  the  individual  elements  and  then  develop  the  integrated 
understanding,  but  such  "mixed  instruction"  does  not  benefit  more  advanced  learners  (Pollock,  et 
al.,  2002). 


Learning  Styles 

Although  school  children  up  through  adults  will  indicate  that  they  prefer  to  learn  in  certain  ways 
and  from  certain  kinds  of  instructional  formats  (so-called  learning  styles),  and  there  is  ample 
evidence  that  people  differ  in  aptitude  for  learning  different  kinds  of  material  and  reasoning  in 
different  ways,  there  is  no  evidence  for  the  interaction  between  these  two  that  a  learning  styles 
hypothesis  would  predict.  That  is,  there  is  no  clear  evidence  that  students  having  one  particular 
learning  style  will  achieve  the  best  educational  outcome  when  trained  using  a  method  that 
matches  their  style,  as  opposed  to  training  that  matches  some  other  learning  style  (Pasher,  et  al., 
2009). 

Types  of  Training  and  Practice:  Mental  Practice 

Given  that  mental  rehearsal  or  imaginary  practice  appears  to  be  an  effective  strategy  to  reduce 
forgetting  or  skill  decay  during  the  retention  interval,  mental  rehearsal  can  be  considered 
"practice.”  Research  has  shown  that  mental  practice  can  serve  as  an  effective  substitute  for 
physical  practice  for  both  enhancing  knowledge  of  particular  sequences  and  improving  general 
skill.  Indeed  mental  practice  might  be  superior  to  physical  practice  under  circumstances  that 
promote  retroactive  interference.  By  hypothesis,  mental  practice  can  retard  forgetting  and 
promote  transfer  of  training  (Healy,  2007). 

Measurement  Issues 

Sequencing,  whether  by  content  or  by  readiness,  has  implications  for  evaluation.  Dodd  (2009) 
presents  a  useful  scheme  for  classifying  military  jobs  in  terms  of  kinds  of  knowledge  required 
and  types  of  decision-action  roles.  For  example,  the  equipment  operator  needs  technical  skills  to 
operate  equipment  and  conceptual  knowledge  to  diagnose  and  repair  problems.  The  unit 
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commander,  in  contrast,  requires  conjectural  skill  and  problem  solving  skill  to  cope  with 
complex  situations. 

Such  cognitively  different  jobs  require  different  forms  of  learning  and  experience,  which  in  turn 
will  be  best  measured  in  ways  that  are  appropriate  to  the  cognitive  work.  For  example,  time 
taken  to  complete  tasks  and  degree  of  confidence  are  useful  training  measures  for  certain  jobs 
and  tasks.  However,  they  may  not  tap  into  the  decision-maker's  approach  to  situation  assessment, 
or  ability  to  spot  anomalies,  or  motivation  to  practice  in  at  unfamiliar  and  unforeseen  situations. 

2.3.2  Problem-Based  Learning  and  Guidance  Fading 

The  educational  philosophy  of  John  Dewey  (1916)  advocated  "learning  by  doing"  and 
emphasized  the  development  of  critical  thinking  skills  through  problem  solving  exercises. 
Originating  primarily  in  the  area  of  medical  education,  Problem-Based  Learning  is  an 
instructional  method  that  has  students  collaborate  in  work  on  challenging  practical  problems  (ill- 
structured,  open-ended,  ill-defined).  Instructional  activities  include  case-based  or  analogical 
reasoning,  causal  reasoning,  story  creating,  concept  mapping  creating  stories;  reasoning  about 
cases;  concept  mapping,  and  others  (see  Hmelo-Silver,  2004;  Hmelo-Silver,  &  Barrows,  2006). 

Research  by  Sweller  (2006)  and  his  colleagues  on  how  students  learn  algebra  suggests  that 
Problem-Based  Learning  is  more  effective  for  later  or  more  advanced  learning  rather  than  initial 
learning.  Indeed,  they  find  that  initial  learning  is  facilitated  if  the  trainee  studies  “worked 
examples”  (Sweller  &  Cooper,  1985;  Cooper  &  Sweller,  1987).  Early  in  the  learning  process, 
learners  may  find  it  difficult  to  process  a  large  amount  of  information  or  assimilate  concepts  at 
odds  with  their  existing  knowledge.  Thus  the  rigors  of  active  problem  solving  may  be  an  issue 
for  initiates.  Once  learners  gain  expertise  the  scaffolding  provided  by  in  Problem-Based 
Learning  helps  learners  avoid  these  issues.  Sweller  calls  this  “guidance  fading.”  Worked 
problems  are  replaced  by  completions  problems,  with  the  eventual  goal  of  solving  problems  on 
their  own  (Sweller,  Van  Merrienboer,  &  Paas,  1998). 

The  fading  notion  is  defined  with  reference  to  a  concept  of  “cognitive  load”  and  the  goal  of 
reducing  the  cognitive  load  of  learners.  This  does  not  directly  accord  with  the  theory  of 
"desirable  difficulties,"  which  asserts  that  any  condition  that  causes  a  desirable  difficulty  during 
learning  facilitates  retention  (Bjork,  1994;  McDaniel  &  Einstein,  2005;  Schneider,  et  ah,  2002). 
Research  has  established  the  finding  that  learning  at  initial  and  intermediate  levels  can  be  faster 
under  easy  conditions  than  under  difficult  conditions;  performance  immediately  after  learning 
will  tend  to  be  better  for  students  who  learned  under  easy  conditions  than  under  difficult 
conditions.  However,  when  testing  is  after  a  delay  or  in  a  slightly  changed  situation,  it  is  often 
found  that  performance  is  better  for  trainees  who  learned  under  difficult  than  under  easy 
conditions.  In  fact,  introducing  desirable  sources  of  difficulty  or  interference  during  training  has 
been  shown  to  be  an  effective  way  to  promote  retention  and  transfer  (Healy,  2007). 

The  fading  notion  also  does  not  accord  with  what  is  known  about  expertise:  The  achievement  of 
high  proficiency  entails  increasing  the  task  difficulty,  not  decreasing  it.  Thus,  a  simple  fading 
hypothesis  is  bound  to  be  incomplete. 

What  might  be  generalized  is  a  notion  of  guidance  changing,  not  guidance  fading:  What 
“counts”  for  difficulty,  and  a  desirable  difficulty,  differs  from  early  versus  more  advanced 
learners.  Sometimes  withholding  information  or  assistance  can  lead  to  defeat  and  sometimes  it 
involves  a  desirable  difficulty  that  helps  the  learner  advance  to  higher  levels  of  capacity.  The 
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"Assistance  Dilemma"  is  for  the  researcher  or  instructional  designer  to  determine  what  is  optimal 
for  a  given  learning  domain  (Koedinger,  et  ah,  2008). 

Long-standing  notions  like  zone  of  proximal  development 
(Vygotsky,  1978)...  or  model-scaffold-fade  (Collins,  Brown,  & 

Newman,  1990)  suggest  that  instructional  assistance  should  be 
greater  for  beginning  learners  and  be  reduced  as  student 
competence  increases.  Such  long-standing  notions  and  the  more 
recent  extensive  experimental  research  on  cognitive  load,  desirable 
difficulties,  etc.  might  lead  one  to  wonder  “what’s  the  dilemma?” 

Just  give  novices  more  assistance  and  fade  it  away  as  they  become 
more  expert.  However,  current  theory  does  not  predict  how  much 
assistance  to  initially  provide  nor  when  and  how  fast  to  fade  it 
(Koedinger,  et  ah,  2008,  p.  2156). 

2.3.3  Feedback 

Research  shows  that  giving  and  withholding  of  assistance  or  feedback  have  both  costs  and 
benefits.  Sometimes  instructional  assistance  can  be  a  crutch  that  does  not  substantively  advance 
the  learner's  capabilities,  sometimes  it  can  be  a  scaffold  that  does.  At  the  most  general  level, 
training  should  balance  giving  and  withholding  information  or  assistance  to  achieve  optimal 
student  learning.  Key  research  results  and  generalizations  concerning  feedback  have  been 
presented  in  many  reports  and  reviews  (e.g.,  (e.g.,  Marton  &  Saljo,  1976).  Some  were  mentioned 
above  in  the  discussion  of  Desirable  Difficulties. 

In  the  literature  on  the  learning  of  simple  tasks,  it  was  sometimes  assumed  that  feedback  is  best 
when  provided  close  in  time  to  performance,  but  this  is  not  necessarily  the  case  at  all.  While 
immediate  feedback  may  improve  performance  in  the  near-term,  it  can  also  have  the  opposite 
effect  on  long-term  performance.  When  learning  a  skill  that  is  knowledge  or  reasoning-intensive, 
the  learner  benefits  from  having  the  time  to  think  back  on  poor  performance  and  cogitate  on  what 
was  done  right,  what  was  done  wrong,  and  what  might  be  done  differently  in  the  future. 
Immediate  outcome  feedback  can  prevent  such  post-task  cogitation  and  thus  not  contribute  to 
accelerated  proficiency  or  the  improvement  of  metacognitive  skills  (thinking  about  one's  own 
reasoning  strategies). 

While  feedback  is  able  to  guide  the  learner  toward  the  correct  behavior,  it  can  also  inhibit 
performance  in  the  long  run  in  other  ways  (Schmidt,  1991).  If  feedback  is  too  frequent,  it  blocks 
important  information  processing  activities.  Furthermore,  if  there  is  too  much  feedback,  it 
becomes  part  of  the  task  itself,  and  no  longer  serves  the  function  of  reflection  and  organization  of 
information. 

It  should  be  noted  in  this  context  that  the  importance  of  feedback  links  in  a  significant  way  to 
what  is  known  about  high  proficiency  (i.e.,  expertise.)  Over  the  past  two  decades,  the  field  of 
Expertise  Studies  has  revealed  a  significant  difference  between  two  kinds  of  domains.  In  Type  1 
domains  it  is  relatively  easy  to  define  expertise  and  find  true  experts.  Examples  are  sports, 
medicine,  and  weather  forecasting.  In  Type  2  domains,  it  is  not  easy  to  define  expert 
performance.  Examples  are  jurisprudence,  economics,  and  psychotherapy.  Type  2  domains  are 
ones  in  which  the  principal  task  goals  involve  predicting  individual  or  aggregate  human  behavior 
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(Shanteau,  1992),  but  they  are  also  domains  in  which  feedback  is  minimal  to  non-existent,  or  is 
often  significantly  delayed.  Domain-dependence  is  a  very  important  consideration  for  any 
training  program  aimed  at  accelerating  the  achievement  of  proficiency. 

Types  of  Feedback 

Most  learning  researchers  and  theorists  have  discussed  the  importance  of  feedback  and  many 
have  highlighted  the  significant  difference  between  outcome  feedback  and  process  feedback.  The 
latter  informs  the  learning  of  such  performance  aspects  as  correctness  or  accuracy  and  the  latter 
provides  information  and  the  learner's  process  and  what  might  have  been  good  or  not  good  about 
it.  Both  forms  of  feedback  can  be  valuable  to  learners.  For  example,  error  analysis  and 
correction,  as  a  form  of  feedback  training,  is  exemplified  by  After  Action  Reviews  (Hunt,  2009). 
It  is  critical  to  recognize  that  feedback  can  be,  and  typically  has  to  be,  a  mixture  of  types  and 
delivery  strategies:  immediate  vs.  delayed,  outcome  vs.  process. 

Corrective  Feedback 

Studies  of  diverse  domains  of  expertise  reveal  a  great  variability  between  domains  (and 
specializations  within  domains)  in  the  extent  to  which  workers  receive  any  feedback,  let  alone 
timely,  high-quality  feedback  (e.g.,  Salas,  Nichols,  &  Driskell,  2007).  In  some  cases,  the  inherent 
nature  of  the  domain  makes  it  impossible  for  the  practitioner  to  receive  timely  feedback  (e.g., 
long-term  weather  forecasting,  intelligence  analysis  for  policy  projection).  This  may  be  one 
reason  why  it  can  take  a  decade  or  more  for  an  individual  to  achieve  expertise. 

Experts  learn  more  from  their  mistakes  than  from  what  they  get  right.  It  has  been  said  that 
apprentices  make  the  same  mistake  twice,  journeyman  make  the  same  mistake  once  and  experts 
work  until  they  never  make  mistakes.  While  this  is  a  point  well  taken,  domain  specialists  who  are 
intrinsically  motivated  often  seek  out  corrective  feedback  that  allows  them  to  perceive  their 
errors  (Sonnentag,  2000).  The  more  highly  experienced  and  motivated  domain  practitioners  seek 
feedback  from  coworkers.  When  everything  works  the  way  it  is  supposed  to,  one  is  less  likely  to 
receive  feedback  about  what  did  not  work  or  what  might  have  been  done  better.  As  such,  experts 
seek  out  corrective  feedback,  feedback  pointing  out  targets  for  improvement.  Cognitive 
Flexibility/Transformation  Theory  suggests  that  feedback  should  help  the  learner  transcend  the 
inclination  to  invoke  a  knowledge  shield  (i.e.,  rationalize  away  a  misunderstanding),  and  un¬ 
learn  concepts  or  notions  that  incorrectly  simplify  their  understanding  of  the  domain. 

Continuous  Feedback 

It  is  possible  for  feedback  to  be  continuous.  That  is,  the  learning  environment  provides  continual 
displays  of  performance  indicators.  This  is  called  “dynamic  assessment.”  Continuous  feedback 
and  monitoring  as  learners  encounter  increasingly  complex  problems,  can  accelerated  the 
achievement  of  proficiency  in  certain  tasks  (Pellegrino  et  ah,  2001;  Frederiksen,  1990;  Lajoie  & 
Lesgold,  1992).  Feedback  also  allows  for  more  active  self-monitoring  and  self-assessment, 
which  can  enhance  the  development  of  proficiency  (Lajoie,  2003;  Bransford  &  Schwartz,  1999; 
White,  Shimoda,  &  Frederiksen,  2000). 

Educational  psychologist  John  Bransford  (2009)  introduced  the  notion  of  “transfer-appropriate 
processing”  in  cognitive  science  (Barclay,  et  ah,  1974;  Bransford  &  Eranks,  1976;  Bransford  et 
ah,  1979).  In  his  recent  studies  of  classroom  learning  he  has  focused  on  expertise  at  teaching  at 
primary  and  secondary  school  levels.  Bransford  and  Schwartz  (2009)  speculate  that  the  best 
conditions  for  the  development  of  expertise  are  those  that  afford  bi-directional  feedback  between 
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teachers  and  learners — where  the  teacher  learns  from  the  students  and  the  students  learn  from  the 
teachers.  They  advocate  the  use  of  simulations,  to  reinforce  they  ability  to  apply  knowledge. 
They  refer  to  "Top  Gun"  kinds  of  training  programs  as  ones  that  seem  to  provide  such 
bidirectional  feedback  and  are  successful — representative,  measurable  tasks  with  record  of 
performance  and  guided  feedback.  Lack  of  feedback  is  argued  the  reason  that  many  professionals 
in  some  domains  (e.g.,  clinical  psychologists)  seem  not  to  improve.  Bransford  and  Schwartz  also 
address  the  problem  of  "inert  knowledge,"  learners  not  being  able  to  apply  what  they  have 
"learned."  He  sites  some  forms  of  learning  that  are  addressed  to  this  problem,  for  example 
Problem-Based  Learning. 

Debriefing  as  a  Form  of  Feedback 

The  comparison  of  training  in  the  military  with  issues  in  sports  is  interesting  for  two  reasons. 
One  is  the  overall  similarity  of  training  approaches  and  the  second  is  the  possibility  of  adapting 
training  methods  used  on  athletics  over  into  military  arenas  (Fiore,  Hoffman  &  Salas,  2008). 

In  competitive  sports  it  is  standard  practice  for  athletes  to  closely  monitor  and  tabulate 
performance  and  to  engage  in  preparatory  behaviors  in  service  of  improvement  and  reflective 
behaviors  in  service  of  diagnosing  good  and  poor  performance.  One  practice  method  that  is 
associated  with  this  in  sports  is  the  technique  of  studying  “game  tapes.”  Teams  spends  hours 
analyzing  their  opponent  (i.e.,  the  team  for  which  they  are  preparing)  as  well  as  hours  discussing 
and  reviewing  tapes  of  their  actual  play  against  that  team.  This,  of  course,  bears  similarity  to  the 
military’s  use  of  the  brief-debrief  cycle  in  which  teams  engage  in  reflexive  activity  and 
systematic  integration  of  preparation,  execution,  and  reflection  (see  Fiore,  Johnston,  &  Van 
Duyne,  2004;  Vashdi,  Bamberger,  Erez,  &Weiss-Meilik,  2007). 

Preparation  involves  pre-task  activities  such  as  planning  through  the  use  of  techniques  such  as 
briefing  visualizations  or  the  use  of  technologies  such  as  mobile  learning  devices  (see  Metcalf, 
2006),  where  initial  expectations  are  created  in  anticipation  of  the  interaction.  Reflection 
includes  post-task  rumination  on  performance  (e.g.,  debriefing),  where  task  feedback  can  be 
administered  to  individuals  and/or  groups  via  after-action  review  technologies  (see  Fiore, 
Johnston,  &  McDaniel,  2005). 

All  branches  of  the  military,  and  organizations  within  the  branches,  have  detailed  protocols  for 
mission  debriefing  intended  to  result  in  “lessons  learned”  (and  not  just  conduct  root  cause 
analysis  of  critical  incidents)  (e.g.,  Hernandez,  2000).  In  U.S.  Air  Force  training  using 
debriefings  there  is  a  set  structure  to  debriefs  (Brown,  2006).  An  example  is  the  five  phases  of  an 
F-16  debriefing  for  a  "flight  lead":  (1)  gathering  data,  (2)  an  "administrative  phase"  in  which  the 
flight  lead  covers  issues  not  main  to  the  mission,  (3)  reconstruction  of  the  mission,  (4)  analysis  of 
"why  questions,"  and  (5)  generation  of  lessons  learned.  The  main  motivation  appears  to  be 
increased  efficiency  and  accuracy  in  the  conduct  of  debriefing  amidst  the  fog  of  war  and  making 
the  debriefing  fast  and  accurate  (Hernandez,  2000).  It  is  not  clear  how  this  might  contribute  to 
making  debriefing  a  learning  experience,  or  might  detract  from  it. 

Brown  (2006)  has  presented  recommendations  that  would  likely  make  debriefs  more  of  a 
learning  experience.  This  paper  proposes  a  detailed,  standardized  procedure  for  conducting 
debriefings,  focusing  on  Debrief  Focal  Points  (DFP).  It  is  prompted  by  the  perception  that 
current  debriefs  can  be  improved.  In  that  regard,  the  author  notes  common  shortfalls  including: 
The  Debrief  Focus  Points  method  is  intended  to  prevent  inaccuracies  in  mission  reconstruction. 
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and  focus  on  the  determination  of  “contributing  factors”  and  root  causes,  and  use  those — in  the 
debriefing  itself — to  generate  instructional  fixes. 

Debriefing  can  rely  on  Instrumented  Debriefing  Systems  (IDSs),  which  use  mocks  of  aircraft 
instrumentation  to  aid  the  pilot  in  the  debriefing  process.  This  idea  emerged  in  the  1970s  and  has 
been  used  for  debriefs  of  large-scale  exercises  as  well  as  debriefs  for  individual  pilot  mission 
debriefs.  Despite  the  growing  fidelity  and  availability  of  IDSs,  most  syllabi  across  the  combat  air 
forces  require  students  to  reconstruct  missions  by  drawing.  Greater  reliance  on  IDS  is 
recommended  (Gaetke,  2009),  to  facilitate  mission  reconstruction  rather  than  forcing  the  pilot  to 
get  “lost  in  the  weeds”  trying  to  correctly  draw  mission  sub  events  (e.g.,  a  turn  radius  based  on 
the  fact  the  fighter  was  16°  nose  low?)  and  also  to  make  the  debriefing  more  of  a 
leaming/training  experience. 

In  conclusion,  there  is  variety  to  debriefing  methods  and  debriefing  goals,  and  not  all  of  these 
can  or  should  be  changed  or  standardized  since  they  are  formed,  in  part,  by  the  urgencies  and 
exigencies  of  combat  situations.  That  being  said,  there  are  suggestions  for  changes  in  debriefing 
procedures  that  might  make  debriefings  more  of  a  learning/training  experience.  Generally,  these 
involve  structured  methods  for  scaffolding  the  process  of  retrospection  and  reconstruction. 


2.3.4  Desirable  Difficulties 

Psychologist  Robert  Bjork  (1994,  1999,  2001,  2009:  Bjork  &  Allen,  1970)  has  emphasized  a 
number  of  useful  generalizations  from  academic  research: 

•  What  counts  as  an  appropriate  instructional  method  depends  on  whether  the  learning  is 
initial  learning  or  is  advanced  and  longer-term. 

•  Learning  benefits  from  errors  with  good  feedback. 

•  Cases  or  problems  that  make  initial  learning  difficult  (e.g.,  learning  a  sequence  in  mixed 
parts,  rather  than  just  sequentially),  can  aid  long-term  learning  and  higher  order  learning, 
such  as  transfer.  Bjork  calls  these  "desirable  difficulties.” 

Examples  include: 

The  finding  that  methods  that  lead  to  good  immediate  reproductive  memory  (such  as  massed 
practice)  may  not  be  well  suited  to  longer-term  retention,  or  transfer. 

The  finding  that  making  task  performance  harder  during  instruction  by  delaying  feedback  can 
lead  to  better  long-term  retention  (see  Schmidt  and  Bjork,  1992). 

The  finding  that  worked  examples  make  performance  easier  during  initial  instruction,  and  greater 
variability  in  example  content  makes  performance  harder  during  early  instruction.  But 
variability-based  early  instruction  also  contributes  to  better  learning  later  on  (Paas  &  Van 
Merrienboer  1994). 

The  notion  of  “desirable  difficulty”  resonates  with  the  method  called  Problem-based  Learning 
(see  Boshuizen,  2009). 

The  ideas  about  “desirable  difficulties”  resonate  with  what  is  known  about  high  proficiency,  in 
particular  the  implications  for  training  and  mentoring  that  derive  from  the  Cognitive 
Flexibility /Transformation  Theory  (see  Section  4.3.1). 
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While  the  concept  of  desirable  difficulties  has  surface  validity,  and  is  a  concept  upon  which  this 
Report  relies,  it  is  not  clear  how  to  operationalize  the  definition  of  desirable  difficulty  and 
distinguish  desirable  from  undesirable  difficulty.  It  is  a  certainty  that  what  constitutes  a  desirable 
difficulty  is  dependent  on  context  and  upon  the  trainee's  level  of  proficiency.  "Training  for 
complex  tasks  is  itself  a  complex  task,  and  most  principles  for  good  instruction  are  contextual, 
not  universal"  (Reigeluth,  personal  communication). 


2.3.5  Mentoring 

The  English  word  "learn"  comes  from  words  meaning  to  lead,  in  the  literal  sense  of  a  track  or  rut 
caused  by  a  wheel.  This  explains  the  origins  of  idioms  such  as  to  get  off  track,  or  in  the  extreme, 
as  in  delerium  (de  <down>  -laeria  <track>).  From  its  origins  describing  wheel  tracks,  laern 
evolved  in  Romance  languages  to  denote  the  process  in  which  one  person  leads  another:  the 
teacher  and  student  relationship.  In  Old  High  German  OLD  High  German,  learn  meant  not  the 
process  in  which  one  person  remembers  information.  Rather,  it  is  a  two-person  relation  meaning 
to  cause  understanding. 

In  any  given  operational  work  environment,  whether  in  the  private  sector  or  the  military,  there 
will  be  individuals  who  span  the  proficiency  continuum,  including  trainees  (apprentices), 
journeymen,  and  experts.  Workers  at  all  levels  are  generally  accountable  for  one  another’s 
actions.  Often  this  takes  the  form  of  mentoring,  which  can  be  done  informally  or  out  of  intrinsic 
motivation,  or  can  be  a  formal  organizational  process  and  requirement  (see  Hoffman,  et  ah, 

2002) .  Workers  devise  coordinated  solutions  that  mix  expert  and  trainee  contributions  (Lajoie, 

2003) . 

It  is  widely  recognized  that  mentoring  is  important  in  organizational  learning  (e.g..  Ford,  2000). 
Workforce  trends  are  raising  the  stakes  with  regard  to  on-the-job  training  and  mentoring. 
Increasing  complexity,  pace  of  change,  pace  of  decision  making,  and  increasing  worker  mobility, 
all  entail  fewer  opportunities  for  mentoring  (Becerra-Femandez,  Gonzalez,  &  Sabherwal,  2004). 
Because  of  these  trends,  national  policy  is  being  formulated  to  support  research  and  development 
in  an  educational  infrastructure  capable  of  producing  the  next  generation  of  the  science  and 
technology  workforce  (see  Committee  on  Science,  Engineering,  and  Public  Policy,  2006).  Even 
if  such  initiatives  are  successful,  the  results  will  be  far  too  distant  to  help  solve  the  immediate 
utilities  workforce  issues. 

In  some  of  the  craft  guilds  of  the  middle  ages,  a  “master”  was  any  practitioner  who  was  qualified 
to  teach,  in  recognition  of  the  special  qualities  that  a  good  mentor  must  possess  (Hoffman,  1998; 
Renard,  1968).  Thus,  even  a  journeyman  could  be  qualified  as  a  master.  The  field  of  Naturalistic 
Decision  Making  has  approached  the  nature  of  a  good  mentor.  In  Sources  of  Power,  Klein  (2003) 
described  the  kinds  of  techniques  that  skilled  mentors  use  and  the  kinds  of  skills  mentors  need  to 
develop  for  diagnosis,  presentation  of  material,  and  setting  the  right  climate  for  learning.  There  is 
some  research  on  what  makes  for  a  good  mentor  in  the  sense  of  coaching  in  sports  (Fiore  & 
Salas,  2008)  and  there  is  a  large  literature  of  what  makes  for  a  good  teacher  (see  Mistrell,  1999; 
Proctor  &  Vu,  2006).  But  little  is  known  about  what  makes  for  a  good  mentor  in  the  context  of 
the  modern  sociotechnical  workplace.  Little  is  known  about  how  to  mentor  individuals  in  the 
progression  from  journeyman  to  expert. 
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Klein  (2002,  2003)  has  observed  mentoring  activities  in  the  job  context.  The  mentor  establishes  a 
relationship  and  environment  that  fosters  learning,  diagnoses  the  reasons  why  the  trainee  is 
struggling,  and  tailors  instruction  to  fit  the  situation.  The  best  mentors  accomplish  all  three.  They 
have  a  repertoire  of  strategies  for  getting  material  across  (e.g.,  demonstrating  the  task  while 
thinking  out  loud,  interrogating  the  trainee  for  the  rationale  behind  actions,  asking  the  trainee  to 
instruct  them,  etc.).  Less  effective  mentors  or  coaches  usually  have  a  very  small  repertoire,  often 
just  a  single  or  “rote  learning”  mode  of  lecturing  to  the  trainee  about  the  procedures  to  be 
followed. 

Hoffman  (1998)  described  the  key  characteristic  of  the  expert  mentor: 

•  The  ability  to  form  rich  mental  models  of  the  learner’s  knowledge  and  skill. 

•  The  mentor  can  anticipate  when  the  learner  will  form  a  reductive  mental  model, 

•  The  mentor  can  anticipate  the  kinds  of  cases  that  will  lead  the  learner  to  form  a  reductive 
model,  the  mentor  can  recognize  the  kinds  of  practice  experiences  that  will  force  the 
learner  to  go  beyond  his  current  reductive  models. 

Studies  of  the  development  of  high  levels  of  proficiency  and  studies  of  training  have  both  shown 
that  mentoring  or  "dynamic  assessment"  has  to  include  opportunities  to  receive  rich  process  and 
outcome  feedback  as  learners  encounter  increasingly  complex  problems  (Glaser  &  Bassock, 
1989;  Frederiksen,  1990;  Lajoie  &  Lesgold,  1992;  Pellegrino  et  ah,  2001).  Enabling  learners  to 
visualize  the  trajectory  of  change  through  models  and  feedback  also  allows  for  more  active  self¬ 
monitoring  and  self-assessment,  which  can  enhance  expertise  development  (Lajoie,  2003; 
Bransford  &  Schwartz,  1999;  White,  Shimoda,  &  Frederiksen,  2000). 

Mentoring  has  not  been  demonstrated  or  studied  at  the  higher  levels  of  proficiency  and 
extraordinary  expertise  (e.g.,  prodigies,  musicians,  etc.).  There  is  some  research  on  what  makes 
for  a  good  mentor  for  coaching  in  sports  (Fiore  &  Salas,  2008),  and  there  is  a  large  literature  of 
what  makes  for  a  good  teacher  (see  Mistrell,  1999;  Proctor  &  Vu,  2006).  But  little  is  known 
about  how  to  find  good  mentors,  or  what  makes  for  a  good  mentor  in  the  context  of  the  modem 
sociotechnical  workplace,  and  little  is  known  specifically  about  mentoring  individuals  from 
journeyman  level  to  expert.  There  is  an  outstanding  practical  need  for  methods  for  rapidly 
identifying  individuals  having  the  most  experience  in  the  role  of  mentor,  and  then  cognitive  task 
analysis  to  reveal  the  reasoning  strategies  of  mentors,  especially  as  they  deal  with  mentoring 
situations  that  are  challenging.  The  motivation  of  experienced  workers  to  pass  along  their  "tough 
case"  knowledge  is  frequently  there,  but  due  to  the  bureaucratic  habit  of  expecting  that 
mentoring  will  succeed,  they  are  often  left  not  knowing  how  to  proceed,  or  even  knowing  that 
their  tough  case  knowledge  is  critical  and  highly  informative  to  apprentices. 

Mentoring  as  a  means  to  share  knowledge  within  organizations  has  been  receiving  more 
attention  in  recent  years  in  the  literature  on  knowledge  management  and  organizational  learning 
(Zachary,  2007).  Recommendations  include  the  need  for  leaders  to  cultivate  a  "mentoring 
culture,"  engage  workers  at  all  levels  in  the  design  and  implementation  of  mentoring  programs, 
create  value  and  incentives  for  mentoring,  and,  importantly,  focus  mentoring  in  insuring  a 
succession  of  key  expertise  (as  opposed  to  the  loss  of  capability).  Organization  leaders  should 
include  mentoring  capacity  as  a  part  of  the  performance  development  plan  for  new  leaders. 
Leaders  themselves  should  serve  as  role  models  for  mentoring.  Of  particular  benefit  for  leaders 
is  that  mentoring  programs  help  keep  them  abreast  of  developments  in  their  workers'  fields  of 
practice:  mentoring  morphs  as  new  knowledge  comes  into  the  business  environment. 
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2.3.6  Scenario-Based  Training 

Scenario-based  training  is  an  idea  related  to  that  of  Problem-Based  Learning,  but  is  a  generalized 
form,  expressing  the  value  of  training  by  use  of  specific  problems,  cases,  stories,  or  scenarios 
(Fletcher,  2009). 

Military  and  civilian  jobs  are  gaining  in  complexity  and  unpredictability  such  that  conventional 
(e.g.,  classroom)  methods  of  training  have  become  ineffective  and  there  is  a  need  to  train  for 
adaptability  and  flexibility  (Shadrick  &  Lussier,  2009).  There  can  be  little  doubt  of  a  consensus 
in  the  research  community  that  training  for  adaptability  and  flexibility  has  to  rely  on  case-based 
methods  of  training  in  which  important  themes  are  explored  in  representative  complex  situations, 
with  feedback. 

Ways  of  Using  Scenarios 

It  is  important  to  distinguish  Distributed  Learning  from  Computer-Based  Training.  Literally, 
both  are  media  through  which  instruction  can  occur.  CBT  is,  we  might  say,  a  more  traditional 
form  of  instruction,  in  which  the  program  controls  the  flow  of  instructional  activities.  In 
Distributed  Learning  permits  practice  "anytime  anywhere,"  but  it  also  is  generally  formed  around 
case  experiences  or  scenarios.  The  learning  experience  is  more  situation-oriented  and  the  learner 
controls  the  flow  of  events  and  experience.  Thus,  Distributed  Learning  is  regarded  as  more 
appropriate  for  high-end  learning  because  it  makes  the  student  responsible  ("active  learning,  or 
"action-initiation")  and  has  potential  to  be  more  motivationally  engaging.  In  addition,  it  is  the 
one  new  instructional  technology  that  is  being  to  bear  on  the  notion  that  it  can  help 
organizations,  including  the  military,  adapt  to  increases  in  the  complexity  and  technology  of 
work  (Kozlowski  &  DeShon,  2005). 

Computer-based  simulations  are  increasingly  being  used  to  teach  abstract  or  conceptual 
information  (Miller,  Lehman,  &  Koedinger,  1999;  Resnick,  1994;  Schank  &  Parrel,  1988).  By 
providing  perceptual  grounding  for  abstract  concepts  they  aid  in  comprehension,  and  the 
interactivity  can  support  a  deeper  understanding  of  the  material  (Goldstone  &  Sakamoto,  2003). 


2.3.7  Virtuai  Reaiity-Based  Training 


It  has  been  demonstrated  that  a  virtual  reality  experience  can  supplement  the  acquisition  of 
declarative  knowledge.  For  example,  Greenstein,  et  al.  (2008)  investigated  the  effect  of  “Second 
Life”  experiences  in  the  learning  of  introductory  material  in  the  physical  and  behavioral  sciences. 
In  learning  about  tsunamis  and  about  schizophrenia,  students  who  completed  both  a  Second  Life 
experience  and  read  a  handout  achieved  higher  test  scores  and  rated  the  learning  experience  more 
engaging  than  when  they  completed  the  handout  alone.  The  Electric  Power  Research  Institute 
(EPRI)  several  years  ago  identified  VR  technology  as  possibly  being  useful,  and  launched  a 
number  of  projects  (Naser  &  Hanes,  2008).  Eor  purposes  of  the  present  Report,  the  most 
interesting  of  these  was  the  creation  of  a  virtual  model  of  an  air-operated  valve  to  support 
maintenance  and  training.  Eor  training,  videos  were  created  showing  animated  disassembly  and 
assembly  procedures.  Also,  step-by-step  disassembly  and  assembly  sequences  were  provided 
requiring  the  user  to  select  the  correct  next  step. 
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The  VR  system  also  played  into  the  process  of  capturing  and  sharing  the  knowledge  of  the  most 
proficient  workers.  The  virtual  valve  model  was  used  for  eliciting  valuable  tacit  (undocumented) 
knowledge  from  experts  as  they  went  through  the  valve  maintenance  procedure.  This  knowledge 
was  hyperlinked  to  valve  parts,  and  could  be  accessed  by  clicking  on  the  displayed  link.  The 
virtual  valve  and  associated  knowledge  could  be  made  available  to  workers  who  could  benefit 
from  its  availability.  In  what  follows,  we  refer  to  this  as  a  “Virtual  Reality-plus-Knowledge” 
model,  or  VR+K  model. 

Another  project  developed  a  VR+K  model  of  a  pressurizer  room  facility.  Virtual  models  were 
created  from  point  clouds  produced  from  laser  scans  of  the  room.  One  model  showed  the  room 
as  scanned  but  with  colors  added  to  surfaces  representing  the  different  radiation  levels  in  the 
room.  The  models  were  used  to  elicit  valuable  knowledge  from  experts  as  they  went  through 
scenarios  for  preparing  the  room  for  a  re- weld  effort.  This  knowledge  was  linked  to  appropriate 
locations  on  the  virtual  models,  and  could  be  accessed  by  clicking  on  the  displayed  hyperlink. 
The  models  were  shown  prior  to  room  entry  to  workers  preparing  the  room  for  the  re- weld  effort, 
and  to  outage  contractors  who  performed  the  re-welding  work.  EPRI  saw  two  highly  significant 
benefits.  Time  for  rework  due  to  worker  errors  would  be  reduced  because  of  more  effective 
training  and  availability  of  expert  knowledge,  and  workers  received  significantly  lower  radiation 
doses  because  efficient  maintenance  entailed  reduced  exposure  time  and  because  they  had  a 
knowledge  of  the  “hot”  area  locations  as  shown  in  the  virtual  model  with  colorized  radiation 
levels. 


It  was  estimated  that  a  worker  trained  with  a  virtual  valve  would 
require  about  five  to  ten  percent  less  time  to  perform  the  actual 
maintenance.  This  is  attributable  to  the  worker  developing  a  better 
mental  representation  of  the  valve  design  and  required  actions. 
Thus,  he  is  able  to  concentrate  on  the  work  to  be  accomplished 
rather  than  where  parts  are  located  and  how  to  disassemble  and 
reassemble  them  correctly  (p.2). 
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Table  2.8  Demonstrated  benefits  of  virtual  reality-based  training. 


Development  Beneeits 

Training  and  Performance  Benefits 

Virtual  models  can  be  created  quickly,  easily,  and 
at  low  cost  from  3-D  CAD  files. 

Virtual  models  and  associated  expert  knowledge 
can  support  job  planning  and  training. 

In  some  contexts,  models  may  be  viewed  in  2.5-D 
on  existing  PCs  and  on  3-D  VR  systems,  although 
3-D  representations  may  not  be  necessary. 

The  virtual  models  were  found  to  be  an  excellent 
tool  for  pre-job  briefings;  the  actual  work  area 
could  be  viewed  in  detail  rather  than  the  abstract 
“bird’s  eye”  view  provided  by  architectural 
drawings. 

Model  portability  supports  its  use  for  just-in-time 
training  wherever  a  worker  is  located  as 
contrasted  with  the  need  for  the  worker  and  mock- 
up  to  be  physically  together. 

Worker  briefing  time  was  found  to  be  about  the 
same,  but  the  virtual  models  provided  workers 
with  more  confidence  of  what  they  would 
encounter. 

These  demonstrations  involved  tool  evaluation  by 
over  100  utilities  workers.  The  findings  support 
the  notion  that  a  merger  of  VR  training  with 
expert  knowledge  elicitation  might  be 
foundational  to  acceleration  methods. 

Workers  asked  more  questions  in  briefings  in 
which  the  models  were  used,  and  contract  workers 
commented  they  had  a  better  understanding  of 
what  they  would  encounter  in  the  room  as 
compared  with  briefings  at  other  plants. 

A  virtual  model  and  associated  resources,  once 
created,  can  be  used  repeatedly  and  the  knowledge 
base  expanded  in  the  future. 

To  the  experts  from  whom  the  knowledge  was 
captured,  the  process  of  knowledge  elicitation 
made  them  able  to  recognize  when  to  provide 
knowledge  during  the  briefings. 

The  main  limitation  that  EPRI  pointed  to  was  that  while  virtual  models  are  useful  for  training, 
they  can  be  of  limited  value  for  planning  tasks  that  are  highly  routinized  or  proceduralized.  This 
is,  perhaps,  to  be  expected.  Indeed,  a  long-term  view  of  the  VR  approach  relates  directly  to  the 
role  of  training  and  proficiency  to  cope  with  unique  events: 

It  is  becoming  difficult  for  the  utilities  to  obtain  needed  contract 
workers  to  support  recovery  from  outages.  It  is  likely  that 
contractors  and  plant  new  hires  in  the  future  will  be  younger  and 
less  experienced.  Virtual  models  and  associated  expert  knowledge 
could  be  important  tools  to  provide  these  people  with  needed 
training  (p.  3). 

A  VR-i-K  model  of  a  facility  would  make  is  possible  for  workers  to  familiarize  themselves  in 
advance  with  equipment  locations,  work  locations,  and  routes  to  and  from  work  locations. 


2.3.8  Training  for  Fiight  Situation  Awareness 

One  of  the  significant  skills  in  many  domains  is  the  “education  of  attention.”  Especially  in  tasks 
or  situations  where  decisions  are  high-stakes  and  must  be  made  under  time  pressure,  it  is 
important  for  the  decision  maker  to  be  able  to  manage  their  attention  and  deliberately  attend  to 
the  significant  features  or  events  in  the  environment.  The  concept  of  attention  retains  its 
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centrality  today  in  communities  of  practice  where  it  has  come  to  be  known  as  "situation 
awareness"  (SA)  (Endsley,  1992). 

Attention  has  been  a  central  concept  in  psychology  since  psychology  emerged  as  a  branch  of 
philosophy  in  the  1800s.  Attention  is  a  bridge  between  sensation  and  memory — what  one  is 
sensing  is  linked  somehow  to  the  concepts  and  categories  residing  in  memory,  allowing  for 
recognitio=on  and  then  fuller  understanding.  This  is  essentially  what  the  theory  of  SA  is  all 
about.  Situational  awareness  is  seen  as  a  bridge  between  information  and  dynamic  mental  models 
of  the  current  situation  (paralleling  the  historically  prior  doctrine  concerning  the  relation  of 
sensation  and  memory). 

SA  has  been  a  topic  for  considerable  research  in  human  factors  and  cognitive  engineering  (see, 
for  instance,  Sohn  and  Doane,  2004;  who  attempted  to  predict  piloting  SA  by  measures  of 
working  memory  capacity).  SA  is  now  used  broadly  to  refer  collectively  to  a  number  of 
phenomena  of  cognitive  work:  mental  model  formation,  sensemaking,  projection  to  the  future, 
and  others  (see  Klein  &  Hoffman,  2008).  Thus  it  is  applied  in  many  work  and  domain  contexts 
and  is  measured  in  a  variety  of  ways  (see  Endsley  &  Garland,  2000).  An  analysis  of  SA  based  on 
the  judgments  of  fighter  pilots  listed  31  component  elements,  ranging  from  tactical  knowledge  to 
planning  ability  (Waag  &  Houck  1994) 

The  origins  of  the  SA  term  and  concept  are  in  military  aviation,  specifically,  the  ability  of  the 
pilot  in  a  dog  fight  to  “get  inside  the  OODA  loop  of  his  opponent.”  (In  a  sense,  training  for  flight 
SA  is  what  Top  Gun  is  all  about;  see  Carroll,  1991)  Traditionally,  pilot  skill  at  “getting  inside  the 
opponent’s  head”  and  anticipating  the  opponent’s  actions  have  been  assessed  by  peers  and 
experienced  pilot  trainers  using  rating  scales  based  on  SA  elements.  It  is  still  so  evaluated,  and  is 
known  to  be  highly  reliable  and  correlates  with  SA  as  measured  in  simulated  combat  missions 
(see  Bell  &  Eyon,  2000). 

The  measurement  of  SA  is  in  some  applications  somewhat  equivalent  to  measuring  performance, 
and  the  “getting  inside  the  opponent’s  head”  feature  of  SA  is  a  defining  feature  that  has  been  lost 
in  most  of  the  applications  of  SA,  which  now  seems  to  refer  to  general  attention  to  cues  and 
patterns  and  dynamics  of  unfolding  events  in  the  world,  not  to  the  attempt  to  reason  about 
someone  else’s  reasoning.  “SA  is  an  abstraction  that  exists  within  our  minds,  describing 
phenomena  that  we  observe  in  humans  performing  work  in  a  rich  and  usually  dynamic 
environment”  (Billings,  1995). 

The  “Eevels  of  SA”  are  a  clustering  of  mental  processes,  most  of  which  were  substantively 
described  in  the  1700s  and  1800s.  SA  Eevel  1  as  described  involves  mental  operations  including 
sensation,  cue  detection,  feature  identification,  monitoring,  awareness  of  states,  perception, 
attention,  and  others.  SA  Eevel  2  involves  perception,  comprehension,  and  interpretation  (“of 
that  portion  of  the  world  that  is  of  concern  to  the  observer”).  Eevel  3  SA  is  called  “projection.” 
Psychologists  of  the  early  eras  referred  to  this  as  “imagination”  and  “apprehension.”  Since  the 
1980s,  it  has  been  known  in  cognitive  psychology  as  “mental  model  formation.”  Eevel  3  SA  is 
conscious  awareness  of  the  ongoing  events  in  the  world  in  terms  of  the  observer’s  knowledge 
(i.e.,  knowledge  of  the  status  and  dynamics  of  the  elements  and  comprehension  of  the  situation). 
Eevel  1  and  Eevel  2  awareness  is  “extrapolated  forward  in  time  to  determine  how  it  will  affect 
future  states  of  the  operational  environment.” 

The  notion  of  level  does  imply  that  events  or  outcomes  at  one  level  feed  into  events  at  the  next 
higher  level.  But  given  the  slipperiness  of  the  categories  of  these  mental  operations,  mental 
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functions  can  be  seen  to  inhabit  more  than  one  Level.  Levels  2  and  Level  3  both  involve  forms  of 
judgment  or  evaluation.  All  of  the  SA  processes  depend  on  memory  and  contact  with  knowledge. 
They  all  involve  some  form  of  understanding,  or  “developing  a  comprehensive  picture  of  the 
world.”  Level  2  SA  is  said  to  involve  the  integration  of  information,  but  that  includes  integrating 
information  to  understand  how  it  will  impact  upon  the  individual’s  goals  and  objectives.  The  use 
of  future  tense  would  seem  to  ascribe  the  mental  process  to  Level  3. 

It  is  sometimes  apparent  when  awareness  has  gone  from  sensory  detection  of  some  sort  of 
occurrence  (What  was  that?),  to  an  interpretation  of  that  occurrence  (It’s  a  dog),  and  thence  to  a 
broader  understanding  of  an  event  (I’d  better  swerve). 

In  measuring  SA,  the  participant  is  placed  in  a  simulated  scenario  and  then  at  random  and 
unpredictable  times  the  scenario  is  briefly  frozen  and  is  asked  a  couple  of  brief  questions.  The 
questions  get  at  various  levels  of  SA:  “What  are  you  seeing?”  What  will  happen  next?”  and  so 
forth.  Endsley’s  seminal  research  (Endsley,  1995)  demonstrated  that  brief  time  freezes  do  not 
substantively  affect  performance  and  do  not  disengage  the  participant  form  the  scenario.  In  the 
domain  of  piloting,  Endsley  (2006)  showed  that  a  measure  of  situational  awareness  is  a  better 
predictor  of  performance  at  simulator  tasks  than  number  of  hours  of  flying  experience. 
Interestingly,  the  concept  of  SA  was  first  proffered  by  fighter  pilots,  who  referred  specifically  to 
the  need  for  a  pilot  to  “get  inside  the  head”  of  the  adversary  in  a  dogfight.  (This  would  be  level  4 
SA.)  (See  Endsley,  1989.) 

Endsley  and  Robertson  (2000)  and  Shebilske,  et  al.  (2000)  reviewed  the  literature  on  SA 
training.  The  levels  of  SA  are  useful  as  a  framework,  for  analyzing  the  frequency  of  errors 
(especially  pilot  errors)  at  the  different  SA  levels,  comparing  error  rates  for  better  and  worse 
performing  pilots  and  aircrews.  Such  an  analysis  is  suggestive  of  what  the  training  needs  might 
be  for  a  pilot  or  team  that  is  performing  at  a  less-than-proficient  level  (e.g.,  practice  at  estimating 
risk  in  certain  situations). 

Endsley  and  Robertson  (2000)  identify  low  SA  as  one  of  the  key  factors  in  aviation  failure. 
Because  it  is  so  important  to  the  success  of  pilots,  methods  for  training  and  evaluation  SA  are 
imperative  to  the  success  of  aviation  training  programs. 

The  authors  posit  that  SA  training  can  be  developed  via  two  main  methods.  The  first  method  is  to 
examine  the  specific  ways  in  which  errors  occur,  based  on  surveys,  records,  and  experiments. 
The  second  method  is  to  examine  how  pilots  maintain  SA,  and  the  differences  between  pilots 
who  do  maintain  SA  and  those  who  do  not.  The  authors  continue  by  providing  examples  from 
the  literature,  citing  studies  that  used  techniques  such  as  the  critical  incidence  method  in  order  to 
determine  how  SA  is  maintained  in  practice. 

Based  on  the  reviewed  studies,  the  authors  provide  a  number  of  methods  by  which  SA  can  be 
improved.  Task  management  skills  allow  pilots  to  deal  with  interruptions  and  workload  that 
would  otherwise  reduce  SA.  The  development  of  comprehension  allows  pilots  to  properly 
determine  the  consequences  of  various  occurrences  during  flight,  while  the  ability  to  project  and 
plan  for  future  events  is  closely  tied  to  workload  and  SA.  Additionally,  individuals  with  high  SA 
engage  in  information  seeking  and  self-checking  actions. 

Given  that  effective  pilots  exhibit  these  skills,  Endsley  and  Robertson  propose  a  number  of 
interventions  to  ensure  the  improvement  of  SA.  Higher-order  cognitive  skills  training  can  be 
used  to  train  skills  necessary  to  the  development  of  SA,  such  as  contingency  planning. 
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information  sharing,  and  filtering.  The  authors  argue  that  training  these  skills  will  also  provide 
increases  in  SA.  Extensive  pre-flight  briefings  also  provide  pilots  with  a  more  comprehensive 
mental  model  of  their  task,  increasing  SA.  During  flight  simulation,  techniques  such  as  the 
Situation  Awareness  Global  Assessment  Technique  (SAGAT)  can  be  used  to  provide  pilots  with 
more  feedback  about  their  own  SA  quality,  engendering  metacognitive  .skills. 

Endsley  and  Robertson  note  an  outstanding  research  need:  Eew  research  programs  have 
attempted  to  train  specifically  for  SA,  across  the  SA  levels  (attention  management,  high-level 
cognitive  skills.  She  notes  that  the  SA  levels  could  be  use  to  structure  feedback.  Related  to 
research  on  sports  (and  the  education  of  perception  and  attention),  Endsley  notes  that  training 
can  emphasize  the  perception  of  important  cues  and  cue  patterns  signaling  when  a  situation  is 
typical  or  is  atypical.  This  fits  similar  recommendations  about  feedback  and  training  from  other 
researchers. 

There  is  one  project  that  has  attempted  SA  training,  a  project  on  the  training  of  teams  of  aircraft 
maintenance  personnel  (Endsley  &  Robertson,  1996).  An  analysis  of  the  “SA  Requirements”  for 
teams  served  as  a  useful  framework  for  delineating  the  training  areas,  including:  shared  mental 
models  of  situations,  failures  to  share  information  across  teams  whose  tasks  interacted,  the  need 
for  the  establishment  of  shared  goals  across  teams.  The  resultant  SA  training  course  was 
administered  at  the  maintenance  bases  of  a  large  commercial  airline.  Eollowing  the  course, 
participants  rated  the  course  as  uniformly  “very  useful”  for  increasing  safety  and  teamwork,  and 
would  result  in  considerable  change  in  how  the  maintenance  teams  operated.  These  ratings  were 
maintained  in  a  one-month  follow-up,  which  the  researchers  interpreted  to  mean  that  there  was  a 
change  in  the  work  and  a  positive  effect  on  performance  and  the  training  was  “highly 
successful.” 

In  applying  SA  notions  to  “human-centered  training,”  Shebilske,  et  al  (2000)  noted  that  one  can 
apply  the  finding  from  cognitive  science  that  deliberative  analysis  characterizes  the  reasoning  of 
trainees  whereas  automaticity  characterizes  the  reasoning  of  high  proficient  practitioners.  The 
implication  of  this  would  be  that  initial  training  should  not  be  overly  dependent  on  automation 
but  rather  should  allow  the  participant  the  opportunity  to  learn  the  procedures  that  might,  in  the 
operational  context,  be  handled  by  the  automation.  This  notion  would  seem  to  contradict  the 
recommendation  that  training  should  emulate  the  operational  situation  to  the  extent  possible,  but 
this  is  a  matter  to  be  resolved  by  research,  specifically  on  the  question  of  to  what  depth  and 
extent  do  trainees  (or  proficient  practitioners,  for  that  matter,  need  to  know  what  the  computer  is 
doing  “deep  down  inside.”  Additional  recommendations  for  training  include  training  to  allow 
practitioners  to  develop  good  mental  models  and  training  in  the  management  of  attention.  Such 
recommendations  would  not  derive  uniquely  from  SA  notions,  of  course  (see  Damos  &  Wickens, 
1980;  Gopher  &  Brickner,  1980). 


2.3.9  Training  for  High  Proficiency 

A  prominent  theme  in  Expertise  Studies  is  that  high  proficiency  stems  from  self-assessment  and 
self-directed  learning,  deliberate  practice,  and  the  self-assessment  of  the  development  toward 
expertise.  These  depend  on  learning  environments  that  promote  progression  of  proficiency 
(Eletcher,  2009).  There  is  relatively  little  research  on  training  at  the  high  end  of  proficiency. 
Interestingly,  the  Ericsson,  et  al.  (2007)  Cambridge  Handbook  of  Expertise  and  Expert 
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Performance  includes  no  discussion  of  training  or  training  issues.  This  links  directly  to  the  core 
problem  motivating  the  present  Report. 

Practice.  Practice.  Practice 

One  of  the  foundational  laws  of  psychology  is  Aristotle’s  law  of  frequency,  that  the  more 
frequent  an  experience  or  association,  the  better  will  be  memory  for  it.  This  idea  spans  the  entire 
history  of  psychology,  leading  to  associative  trace  strength  theories  of  memory,  theories  of 
memory  decay,  and  other  notions  and  research  paradigms.  The  law  of  frequency  is  arguably  true, 
and  applies  as  well  to  the  learning  of  experts  as  it  does  to  the  learning  of  the  newborn. 

In  sports,  the  majority  of  a  performer’s  time  engaged  in  the  task  is  preparation.  Sports  performers 
often  engage  in  competition-specific  preparation  which  allows  them  to  prepare  in  such  a  way  that 
mirrors  the  competition  and  sufficiently  adapt  to  its  contextual  features  (Eccles,  et  al,  2009; 
Vicente,  2000).  A  general  finding  from  studies  of  expertise  broadly  is  that  the  amount  of  practice 
is  the  single  greatest  determinate  of  superior  performance,  followed  by  specificity  of  practice  and 
motivated  of  practice  at  difficult  tasks  (Schmidt  &  Lee,  2005;  Ericsson,  2003). 

Motivation 

A  research  finding  from  many  studies  spanning  decades,  is  that  high  proficiency  is  achieved  only 
after  years  of  sustained  and  motivated  effort,  which  Thorndike  (1912)  called  “practice  with 
zeal.”  The  phenomenon  has  often  been  noted  (Bereiter  &  Scardamalia,  1993;  Ericsson,  Charness, 
Eeltovich  &  Hoffman,  2006;  Hoffman  &  Militello,  2008).  The  core  idea  has  even  made  its  way 
into  the  popular  press  in  stories  told  about  the  nature  of  talent  versus  ability  (e.g.,  Gladwell, 
2009;  Shenk,  2010).  “Amateurs  work  until  they  get  it  right;  professionals  work  until  they  can't 
get  it  wrong”  (General  Motors  Corporation  television  commercial,  August  2006).  “Apprentices 
make  the  same  mistake  twice;  experts  make  the  same  mistake  once”  (Guillermo  Alvarado, 
2006).  The  central  implication  for  training  is  the  need  to  emphasize  motivational  factors. 

The  role  of  motivation  in  achieving  high  proficiency  is  highlighted  in  the  literature  on 
organizational  learning  (e.g..  Center  for  Applied  Systems  Studies,  2010).  This  is  referred  to  by 
the  notion  of  the  “reflective  practitioner”  (Schon,  1983)  who  can  improvise  or  create  hybrid 
options  if  an  existing  set  of  options  seems  to  be  inappropriate.  The  reflective  practitioner  gains 
this  capacity  by  analyzing  their  own  ability  to  deal  with  situations. 

Trainees  need  to  have  a  sense  of  achievement  to  balance  a  sense  of  failure,  which  might  be  more 
impactful  because  the  work  is  cognitively  difficult  and  hence  trainees  might  be  especially  prone 
to  failure  in  the  early  phases  of  instruction.  "Research  indicates  pervasive  effects  for  the 
motivational  components  on  self-regulatory  processes,  learning,  and  performance"  (Kozlowski  & 
DeShon,  2005,  p.  6-7). 

Ericsson's  "expert  performance  approach"  translates  these  ideas  into  the  concept  of  “deliberate 
practice”  (see  Ericsson,  et  ah,  1993).  The  approach  consists  of  two  ideas.  Eirst,  it  defines 
expertise  as  superior  performance  at  representative  tasks.  Second,  it  asserts  that  claims  about 
expertise  can  only  be  made  with  reference  to  individuals  who  have  been  empirically 
demonstrated  to  show  reliably  superior  performance  under  controlled  circumstances.  These 
operational  definitions  are  important  from  the  standpoint  of  academic  research. 

Problems  with  this  “approach”  include  the  following: 
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•  It  does  not  adapt  to  application  situations  where  the  expert's  performance  cannot  be 
brought  under  laboratory  control. 

•  It  does  not  adapt  to  domains  in  which  the  principal  tasks  (and  hence  the  performance 
measures)  are  ill- structured  and  problem  solutions  are  well-defined. 

•  It  does  not  adapt  to  domains  in  which  tasks  are  always  changing. 

•  It  does  not  adapt  to  domains  in  which  there  are  only  a  few  “experts.” 

•  It  is  mute  with  regard  to  proficiency  scaling,  apart  from  dividing  the  world  in  to  people 
who  are  experts  and  people  who  are  not. 

•  It  ignores  additional  and  important  elements  within  the  conceptual  definition  of  expertise, 
specifically,  that  experts  can  perform  well  at  tough  cases,  and  can  create  new  methods 
and  strategies  "on  the  fly." 

•  It  tacitly  devalues  the  study  of  expert  knowledge, 

•  It  tacitly  devalues  the  study  of  studying  expert  reasoning  strategies. 


Ericsson  would  likely  disavow  the  last  of  these  assertions,  since  studies  using  this  “approach” 
have  revealed  strategies  (e.g.,  strategies  for  managing  working  memory  load  on  the  part  of  expert 
memorizers  of  long  strings  of  digits).  But  the  “approach”  as  stated  reflects  its  origins  in  the 
paradigm  of  academic  research  on  human  performance,  rather  than  the  paradigm  of  cognitive 
work  analysis  or  human  factors.  Our  reason  for  calling  out  the  ostensive  limitations  of  the 
“approach”  is  that  if  expertise  is  achieved  only  after  deliberate  practice  (the  "10-year  rule  of 
thumb),  then  acceleration  would  not  be  possible. 

Training  for  Adaptivity 

Kozlowski  and  DeShon  (2005)  reported  a  series  of  studies  on  how  individuals  and  teams  adapt, 
including  an  analysis  of  interventions  intended  to  enhance  adaptive  capacity.  They  present  a 
model  in  which  adaptive  capacity  is  said  to  depend  on  the  following  factors: 

•  Goal  representation.  How  individual  and  teams  conceive  of  their  goals  will  influence 
their  learning  and  adaptation  strategies. 

•  Self-regulation.  Complex  cognitive  work  requires  self-regulation  across  multiple  tasks 
and  goals,  especially  in  teamwork  contexts. 

•  Resource  allocation.  How  can  individuals  and  team  be  trained  to  optimally  allocate 
resources  across  multiple  asks  and  goals? 

Instruction  that  prompts  trainees  to  approach  learning  as  an  effortful  and  mindful  process 
("active  learning")  are  useful  in  training  for  complex  cognitive  work,  generally  associated  with 
improved  skill  acquisition  and  increased  adaptivity.  Trainees  need  to  think  about  their  own 
strategies  (“mindfulness  training”).  Trainees  need  to  focus  on  mastering  tasks,  rather  than  getting 
individual  problems  right  ("mastery  training").  To  counterpoint  the  emphasis  on  the  role  of 
practice,  we  hasten  to  point  out  that  the  basic  aptitude  or  intelligence,  and  the  background 
knowledge  of  the  learner  certainly  place  some  bounds  on  training  effectiveness  and  outcome.  A 
study  that  reviewed  measures  and  results  found  for  predicting  on-job  performance  in  the  U.  S. 
military  from  1952-1980  (Vineberg  &  Joyner,  1982)  found  that  job  knowledge  tests  (of  prior 
knowledge)  were  a  useful  predictor  of  performance,  that  aptitude  was  the  most  utilized  predictor 
in  the  design  of  training  plans,  and  that  supervisor  ratings  were  the  most  used  criterion,  useful  in 
jobs  where  social  skills  were  important. 
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Interestingly,  the  Vineberg  and  Joyner  report  anticipated  current  concerns.  The  authors  found  a 
need  to  measure  or  evaluate  skill  at  responding  to  “situational  requirements”  which  we  would 
refer  to  as  adaptive  capacity.  Also,  the  authors  recommended  training  and  testing  based  on 
"miniature  training  and  assessment  centers,"  and  the  use  of  self-studying  as  a  measure,  reflecting 
perhaps  the  notion  of  deliberate  practice.  While  in  the  present  Report  we  have  emphasized 
training  by  scenarios  and  simulations  as  a  means  for  acceleration,  Vineberg  and  Joyner  (1982) 
saw  it  as  a  way  to  assess  the  initial  competence  level  of  trainees  and  predict  their  performance  or 
success  following  training: 

At  least  two  approaches  to  prediction  would  appear  to  take  some 
advantage  both  of  the  point-to-point  strategy  and  of  the  efficiency 
of  training  performance  as  predictor:  (1)  the  use  of  miniaturized 
training  and  assessment  centers  in  which  prospective  trainees  can  be 
tested  in  a  sample  of  work  activity,  and  (2)  increased  focus  on 
individualized,  self-paced  training  as  a  predictor.  If  the  time  for 
training  and  observation  in  the  center  were  kept  brief,  the  former 
approach  could  be  used  for  entry  screening  for  military  service 
(performance  in  self-paced  training  could  not,  of  course,  be  used  for 
entry  screening)  (p.24). 

Psychological  Support  Skills 

High  proficiency  in  all  domains  involves  meta-cognitive  skill,  as  captured  in  the  notion  of  the 
“reflective  practitioner.”  The  training  of  self-reflection  is  referred  to  as  Psychological  Support 
Skills  Training.  Is  involves  practice  at  mental  rehearsal,  self-talk  (that  is,  mindfulness)  goal¬ 
setting,  and  goal  focus  versus  performance  focus  (Eccles  &  Feltovich,  2008). 

These  skills  can  be  trained  and  learning,  for  instance,  trainees  can  be  taught  to  counter  negative 
self-talk,  that  is,  to  recognize  when  their  self-talk  is  critical  and  self-defeating,  and  focus  instead 
on  looking  for  cues  to  positive  action.  Eccles  and  Eeltovich  speculate  that  "Novices  who  learn 
and  apply  psychological  support  skills  experience  accelerated  learning"  (p.  53). 

Source  of  Instructional  Materials 

One  source  of  materials  for  accelerated  training  will  be  scenarios,  as  discussed  above  in  section 
2.3.6.  As  we  asserted  in  the  introduction  to  this  Section  of  this  Report,  expertise  is  the  “gold 
standard”  for  training,  it  is  the  source  for  training  materials,  and  it  is  a  goal  for  accelerated 
learning.  A  consensus  in  the  human  factors  and  cognitive  engineering  communities,  is  that 
Cognitive  Task  Analysis  for  specifying  what  knowledge,  strategies,  skills,  and  attitudes  is 
required  for  conducting  a  tasks,  and,  hence  is  required  for  the  formulation  of  training  and 
instruction  (Schraagen,  2009).  Indeed,  cognitive  task  analysis  emerged  in  instructional  design 
research  (for  the  military)  (see  Hoffman  &  Militello,  2009). 


2.3.10  Conclusion 

The  fact  that  learning,  training  and  instructional  design  are  so  highly  complex,  with  multiple 
factors  that  interact,  makes  if  very  difficult  to  compose  anything  like  a  recipe  for  instructional 
design  for  proficiency  training.  Indeed,  the  complexity  is  so  daunting  as  to  bring  some  to  near 
despair.  For  example,  Dodd  (2009)  discussed  the  implications  of  moving  target  problems  for 
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military  training  and  "end-to-end"  evaluation.  In  an  ideal  situation,  one  would  be  able  to  trace  the 
effects  of  instruction,  from  the  very  beginning,  noting  its  effect  across  the  entire  span  of  learning 
various  knowledge  and  skills,  right  through  to  final  operations  performance.  Reasons  why  this  is 
not  possible  are: 

•  The  time  taken  for  the  changes  due  to  training  to  percolate  through  to  operational  effects 
can  be  long  and  involves  complex  interactions  with  other  functional  attributes,  such  as 
organizational  constraints  and  operational  pressures. 

•  There  is  no  straightforward  track  through  the  formal  process  of  lessons-learned  enabling 
us  to  give  credit  to  any  particular  input  (or  lack  of  input)  for  any  specific  observed 
outcome. 

•  Operational  outcomes  are  so  contextually  and  circumstantially  driven  that  they  defy 
backwards-looking  interpretation  necessary  to  understand  cause-and-effect  linkages. 

•  Operational  effectiveness  outcomes  are  tied  to  scenarios  making  it  impossible  to  get 
representative  coverage  permitting  generalizations  that  specify  which  training  inputs  are 
effective  in  meeting  which  possible  or  future  operational  challenges. 

"So  we  are  often  reduced  to  stating  the  obvious:  that  doing  no  training  or  education  would  result 
in  poorer  operational  outcomes"  (Dodd,  2009). 

Others  are  more  hopeful,  especially  with  regard  to  a  vision  for  acceleration. 

Further  discussion  of  training  to  high  proficiency  appears  later  in  this  Section,  in  subsection  2.7, 
reviewing  the  existing  examples  of  success  at  accelerated  learning.  In  the  Section  6.0  that 
concludes  this  Report,  we  will  carry  over  the  positive  generalizations  that  can  be  confidently 
asserted  based  on  the  literature  on  training  that  has  been  reviewed  in  this  Section. 


2.4.  Transfer 


The  major  theory  of  transfer  is  the  "common  elements"  theory.  Based  on  this  idea,  one  would  say 
that  training  should  minimize  the  transfer  distance  from  training  to  workplace.  Recent  research  is 
suggestive  of  the  conditions  that  promote  transfer,  such  as  the  judicious  use  of  particular  kinds  of 
feedback  (Schmidt  &  Bjork,  1992).  However,  performance  issues  go  beyond  transfer  from  the 
classroom  to  the  operational  context.  Simply  “working  at  a  job”  does  not  promote  progression 
along  the  proficiency  continuum.  Unless  there  is  continuous  deliberate  practice  at  difficult  tasks, 
the  only  thing  one  can  do  “on  the  job”  is  forget  and  actually  experience  degradation  of  skill. 

Furthermore,  the  current  challenges  for  military  training  involve  two  different  sorts  of  transfer. 
One  is  transfer  across  mission  types.  An  example  would  be  an  infantry  commander  who  knows 
traditional  warfare  who  is  asked  to  develop  tactics  for  an  insurgency  operation.  The  second 
challenge  is  transfer  across  responsibilities.  An  example  would  be  a  Warfighter  skilled  at  F-16 
engine  maintenance  who  is  promoted  to  a  supervisory  position.  Since  different  skill  sets  would 
be  involved,  one  would  need  to  train  for  the  new  role,  and  not  just  assume  transfer  would  or 
might  happen.  Motivation,  intelligence  and  other  mediating  factors  would  be  enabling  conditions 
perhaps,  but  not  guarantees  of  transfer.  The  supervisory  position  might  be  in  the  area  of  aircraft 
maintenance,  but  that  too  would  be  an  enabling  condition  and  not  a  guarantee  of  transfer. 

Transfer  of  learning  or  training  has  been  a  salient  issue  in  recent  discussions  of  national 
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educational  policy,  given  its  direct  pertinence  to  the  complexification  of  the  modem  workplace, 
the  emergence  of  a  workforce  which  does  not  define  itself  in  terms  of  careers  with  single 
employers,  and  the  general  need  for  a  workforce  that  is  capable  of  critical  thinking  (see  Haskell, 
2001).  In  other  words,  for  the  economy  and  for  organizations  to  be  competitive,  the  workforce 
must  be  capable  of  transfer  of  learning  (Broad  &  Newstrom,  1992). 

There  are  a  number  of  different  meanings  of  "transfer"  in  the  literature.  Some  modern 
researchers  refer  to  the  transfer  of  a  single  acquired  skill  from  the  training  context  to  the 
performance  context.  This  meaning  of  "transfer"  is  a  reference  to  the  durability  of  training  or  the 
retention  of  some  skill  or  level  of  capability,  e.g.,  how  long  the  learner  can  accurately  retain  and 
utilize  information.  Many  studies  that  refer  to  transfer  compare  performance  in  a  learning  context 
to  performance  in  a  subsequent  evaluation  context,  when  the  tasks  or  problems  remain 
essentially  the  same.  This  meaning  of  "transfer"  is  rather  the  same  as  the  notion  of  skill  retention. 
For  transfer  in  this  sense  to  be  effective,  “the  training  must  incorporate  the  complete  set  of 
transfer  task  requirements,  including  any  secondary  task  requirements  imposed”  (Healy,  et  ah, 
2005,  p.  11). 

Research  on  transfer,  in  the  sense  of  durability  or  retention  has  resulted  in  some  useful  general 
findings.  One  is  the  phenomenon  of  "desirable  difficulty"  which  was  discussed  above  in  this 
Report  in  Section  2.  Things  that  make  learning  difficult  facilitate  transfer  (in  the  sense  of 
retention  of  capability)  to  a  new  task  as  well  as  long-term  retention  of  the  original  task.  By  the 
training  difficulty  principle,  any  condition  that  causes  a  desirable  difficulty  during  learning 
facilitates  later  retention  and  transfer  (Bjork,  1994;  McDaniel  &  Einstein,  2005;  Schneider, 
Healy,  &  Bourne,  2002).  Another  finding  is  that  for  optimal  performance,  the  entire 
configuration  of  task  requirements  during  training  needs  to  match  those  in  the  transfer  context  as 
closely  as  practically  possible.  By  the  specificity  of  training  principle,  retention  and  transfer  are 
depressed  when  conditions  of  learning  differ  from  those  during  subsequent  testing  (Healy, 
Wohldmann,  Sutton,  &  Bourne,  2006). 

In  this  literature,  learning  “is  assumed  to  refer  to  that  set  of  processes  occurring  during  the  actual 
practice  on  the  tasks  of  interest,  as  assessed  by  performance  measures  taken  at  that  time,  whereas 
retention  is  seen  to  involve  the  set  of  processes  that  occur  after  practice  is  completed,  during 
some  retention  interval,  and  prior  to  a  retention  test”  (Schmidt  &  Bjork,  p.209).  Schmidt  and 
Bjork  argue  that  retention  should  be  viewed  as  a  measure  of  the  effectiveness  of  learning  and  not 
as  a  phenomenon  of  transfer. 

Haskell  (2001)  has  specified  a  number  of  different  meanings  of  "transfer."  These  are  listed  in 
Table  2.8. 
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Table  2.9  Different  kinds  or  senses  of  transfer  (after  Haskell.  2001). 


Content-to-Content 

Applying  knowledge  in  one  content  domain  to 
aid  the  learning  of  knowledge  in  some  other 
content  domain. 

Procedure-to-Procedure 

Using  procedures  learned  in  one  skill  area  to 
work  problems  in  some  other  skill  area. 

Declarative  knowledge-to-Procedural 
knowledge 

Book  knowledge  or  knowledge  of  concepts 
and  principles  learned  about  an  area  aids  in  the 
learning  of  skills,  strategies  or  procedures  in 
that  same  area. 

Procedural  knowledge-to-Declarative 
knowledge 

Experience  with  the  skills,  strategies  or 
procedures  in  an  area  aids  in  the  learning  of 
conceptual  knowledge  in  that  same  area. 

Transfer  of  Self-aware  Strategic  knowledge 

Knowledge  about  one's  own  reasoning  or 
strategies  is  applied  from  the  original  domain 
of  learning  to  some  other  domain. 

Cross-subdomain  transfer  of  declarative 
knowledge 

Knowledge  of  a  concept  or  cause  is 
generalized  across  subdomains  (e.g.,  lightning 
as  a  form  of  electricity). 

Cross-subdomain  generalization  (also  called 
vertical  transfer) 

Knowledge  of  a  particular  is  generalized  across 
subdomains  (e.g.,  learning  about  the  cause  of  a 
particular  war  facilitates  the  learning  about  the 
causes  of  war  in  general). 

Cross-domain  transfer  of  declarative 

knowledge 

This  includes  transfer  based  on  analogy.  It  also 
includes  the  "usefulness  of  useless 
knowledge,"  as  in  trivia  quizzes. 

Lateral  transfer  of  skill 

Learning  one  perceptual  motor  skill  facilitates 
the  learning  of  some  other  very  similar 
perceptual  motor  skill  (e.g.,  ice  skating-roller 
skating). 

For  the  purpose  of  this  Report,  it  is  sufficient  if  we  consistently  distinguish: 

•  Transfer  of  knowledge  (in  the  sense  of  content  knowledge) 

•  Transfer  of  perceptual-motor  skills,  and 

•  Transfer  of  reasoning  strategies. 

Equally  important  is  to  separate  transfer  from  retention,  thus  we  define  transfer  as: 

When  knowledge  skill  or  strategy  learned  in  one  context  or  learned 
for  one  problem/task  type  is  successfully  utilized  in  learning  or 
performance  in  some  other  context  or  for  some  other  problem/task 
type,  where  either  the  contexts  or  the  problem/tasks  types  are  both 
similar  and  different. 

There  is  ample  evidence,  both  scientific  and  personal,  that  transfer  does  happen.  For  instance, 
one  participant  at  the  Working  Meeting  said:  "Preparing  for  the  B-1  after  flying  the  B-52.  Not  a 
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great  jump.  Right  after  my  quals  1  was  told  1  would  be  an  instructor!  There  are  also  experiences 
of  negative  transfer.  In  a  study  of  automobile  driving  skill  of  Americans  driving  in  the  United 
Kingdom,  reports  from  drivers  typically  referred  to  phenomena  of  negative  transfer — stories 
about  how  hard  it  was,  the  problems  experienced,  etc.  The  actual  data  on  performance  showed 
positive  transfer,  however  (Singley  &  Anderson,  1989).  (In  a  formal  demonstration  one  would 
have  to  show  that  the  Americans  achieved  a  base  level  of  acceptable  performance  in  less  time 
than  it  had  taken  for  them  to  initially  achieve  that  level  when  learning  to  drive  in  the  U.S.) 

There  is  also  surprising  evidence  that  transfer  does  not  happen.  Shanteau  et  al.  (2002)  conducted 
an  evaluation  of  the  performance  of  highly  expert  livestock  judges  who  rate  animals  on  such 
dimensions  as  breeding  quality  and  meat  quality.  The  researchers  found  that  skill  at 
discrimination  for  swine  does  not  transfer  to  skill  at  discrimination  for  cattle  (and  vice  versa). 
“Expertise  is  spectacularly  narrow.”  The  surprise  here  is  perhaps  a  reflection  of  the  naive  and 
wrong  assumptions  that  we  non-experts  make  in  apprehending  the  task  without  benefit  of  the 
experts’  skill  acquired  through  perceptual  learning. 

In  retrospecting  about  the  history  of  research  on  training  across  his  own  career,  especially  with 
respect  to  his  classic  book  Training  in  Business  and  Industry  (McGehee  &  Thayer,  1961),  Paul 
W.  Thayer  (2000)  noted  that  "We  have  gone  from  a  simple  view  of  the  learning  and  transfer  of 
training  processes  to  a  much  more  complex  one."  (p.l).  He  noted  that  earlier  work  on  training 
considered  only  single  and  relative  simple  tasks,  and  considered  only  simple  contrasts,  such  as 
that  between  part  and  whole  training,  and  that  between  massed  versus  distributed  practice.  A 
distinction  was  not  made  between  things  that  might  enhance  or  inhibit  transfer  during  training, 
things  that  might  enhance  or  inhibit  transfer  from  training  to  the  job,  and  things  that  might 
enhance  or  inhibit  transfer  on  the  job.  He  acknowledged  that  some  things  they  postulated  were 
simply  wrong.  For  instance,  while  the  notion  of  overlearning  was  considered,  it  has  only  become 
clear  in  years  since  the  1960s  that  overlearning  is  a  prime  determinant  of  decay.  Certainly  off  the 
radar  in  1960  was  team  training  and  the  additional  layers  of  complexity  this  adds  to  the  topic  (see 
Cannon-Bowers,  Tannenbaum,  Salas,  &  Volpe,  1995). 


2.4. 1  Demonstrating  Transfer:  Simple  Perceptual-Motor  Tasks 

Much  of  the  research  by  Robert  Proctor  and  his  colleagues  has  focused  on  transfer  of  newly 
acquired  associations  in  tasks  that  require  a  rapid  choice  response  to  a  stimulus  (Proctor  &  Vu, 
2006).  In  the  prototypical  procedure,  a  two-choice  reaction  task  of  pressing  left  and  right 
response  keys  mapped  incompatibly  to  stimuli  in  left  or  right  locations  (e.g.,  press  “left”  key  to  a 
stimulus  that  appears  on  the  “right”)  is  first  performed.  This  is  followed  after  a  delay  by 
performance  of  another  two-choice  reaction  task  in  which  subjects  respond  to  a  non-spatial 
stimulus  attribute  (e.g.,  color;  the  Simon  task)  and  stimulus  location  is  irrelevant.  Thus,  the 
spatial  mapping  used  in  practice  is  task-irrelevant  in  the  transfer  task.  Without  such  practice, 
responses  are  faster  and  more  accurate  when  the  stimulus  and  response  locations  correspond, 
which  defines  the  Simon  effect.  However,  after  performing  the  incompatible-mapping  task,  the 
advantage  of  the  spatially  compatible  responses  in  the  Simon  task  is  eliminated.  This  outcome 
implies  that  participants  acquired  incompatible  stimulus-response  associations  and  transferred 
them  to  a  subsequent  task  even  though  they  were  no  longer  relevant. 
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Through  manipulating  several  variables,  the  researchers  have  found  that: 

•  The  incompatible  stimulus-response  associations  are  retained  for  at  least  a  week. 

•  The  strength  of  the  associations,  as  implied  by  the  magnitude  of  the  transfer  effect, 
increases  as  the  number  of  practice  trials  increases. 

•  These  associations  have  both  modality  specific  and  general  components,  as  practice  with 
auditory  stimuli  produces  some  transfer  to  a  visual  Simon  task. 

•  With  extended  practice,  more  abstract  rule-like  procedures  are  acquired  that  generalize, 
for  example,  across  different  spatial  orientations. 

•  Practice  with  left  and  right  pointing  arrows  transfers  to  stimuli  varying  in  left  and  right 
locations,  and  vice  versa,  implying  reliance  on  common  visuospatial  codes. 

•  Arrows  also  show  some  indication  of  producing  transfer  to  location  words  (and  vice 
versa)  after  more  extended  practice,  whereas  spatial  locations  do  not,  suggesting  that 
arrows  also  activate  verbal-spatial  codes  common  to  verbal  stimuli. 

•  No  reduction  in  transfer  occurs  when  perceptual  features  of  response  devices  used  in  the 
practice  and  transfer  sessions  are  altered,  but  significant  reduction  occurs  when  responses 
are  executed  differently  (by  pressing  keys  with  two  hands  or  moving  an  index  finger  to  a 
response  key)  in  practice  and  transfer  sessions.  These  results  suggest  that  motoric  or 
procedural  components  are  more  salient  than  perceptual  components. 

Basic  tasks  of  this  type  can  be  used  to  investigate  and  test  characteristics  of  associative  learning. 
The  results  can  be  used  to  develop  and  test  quantitative  and  computational  models  of  skill 
acquisition,  retention,  and  transfer  for  predicting  training  effectiveness. 


2.4.2  Demonstrating  the  Failure  to  Transfer:  The  Domain-Specificity 
of  Knowledge  and  Skill 

Just  as  there  are  studies  showing  positive  transfer,  there  are  studies  showing  that  declarative 
knowledge  does  not  transfer  across  domains  (except  in  analogical  reasoning)  (Haskell,  2001).  A 
number  of  studies  have  shown  that  experts'  knowledge  but  also  their  skills  are  domain-bound 
(e.g..  Beach,  1988;  Chiesi  &  Evans,  1988;  Groen  &  Patel,  1988;  Spilich,  et  ah,  1979).  Thus,  for 
example,  a  study  of  chefs  found  that  they  could  easily  do  the  math  needed  to  convert  a  recipe 
designed  to  feed  12  into  a  recipe  designed  to  feed  six,  but  could  not  do  so  well,  using  the  same 
basic  mathematical  operations,  to  convert  the  prices  of  meals. 

While  an  expert  might  have  highly  refined  and  differentiated  categories  of  concepts  within  the 
domain,  their  knowledge  of  things  outside  the  domain  is  not  so  broad  or  differentiated  (Chase, 
1983;  Chiesi,  et  ah,  1979;  Glaser,  1987;  Scribner,  1984).  In  addition,  the  strategies  of  expert 
reasoning  are  shaped  by  the  tasks  and  contexts  that  are  involved  in  the  domain  (Greeno,  1978; 
Scribner,  1984).  Thus,  for  instance,  performance  at  card  games  by  highly  practiced  individuals 
can  be  significantly  disrupted  by  even  minor  changes  in  the  game  rules  (Sternberg  &  Frensch, 
1992). 
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2.4.3  Promoting  Transfer  by  Exposure  to  Variants 

Task  variability  during  training  can  produce  interference  and,  thus,  retard  initial  skill  learning, 
but  often  enhances  ultimate  performance  (see  Healy,  2007).  It  is  well  established  that  the 
learning  curve  during  training  (that  is,  time-to-criterion)  can  be  affected  if  the  practice  involves 
experience  with  a  variety  of  kinds  of  tasks,  problems  or  test  cases,  sampling  representative  tasks 
and  also  capturing  the  range  of  problems  or  test  cases  on  the  most  important  dimensions, 
especially  difficulty.  The  ultimate  benefit  for  variable  training  is  found  even  when  the  task  used 
in  testing  differs  from  those  used  during  variable  practice  and  is  the  same  as  that  used  during 
constant  practice.  Healy  (2007)  cites  a  study  in  which  children  practiced  the  tossing  of  beanbags. 
Practice  that  varied  the  target  distance  led  to  better  accuracy  at  test  on  an  intermediate  target 
distance  than  did  practice  that  was  restricted  to  that  intermediate  target  distance. 


2.4.4  Motivation  and  Transfer 

It  has  been  demonstrated  that  incentives  can  have  a  strong  effect  on  transfer.  Brennan,  et  al. 
(2009)  examined  whether  transfer  of  training  would  be  improved  by  incorporating  case 
variations  during  the  initial  training.  This  study  examined  whether  transfer  of  training  would  be 
improved  by  incorporating  various  during  the  initial  training.  Participants  were  given  a  visual 
search  task  of  inspecting  luggage  for  weapons.  Participants  were  shown  a  series  of  objects  to 
distinguish  weapon  from  non-weapon  (signal  from  noise).  Participants  were  broken  down  into 
five  groups,  those  that  were  given  positive  “hit  sensitive”  incentives  (more  points  for  correct 
target  identification),  negative  “miss  sensitive”  incentives  (deductions  for  incorrect  target 
identification),  “equal  costs”  (equal  points  for  hits  and  misses),  no  incentives  (no  points 
awarded),  and  a  control  group  that  was  self- trained  and  did  not  receive  initial,  guided  training 
and  were  not  primed  by  specific  targets  from  the  experimenters.  Participants  were  measured  on 
their  “hit  rate”,  “false  alarm  rate”,  and  their  confidence  in  choices.  In  turned  out  that  the  “control 
group”  of  self-trained  participants  showed  superior  performance,  demonstrating  the  highest  hit 
rate  and  the  lowest  false  alarm  rate.  They  also  had  lower  confidence  on  false  alarms  and  higher 
confidence  on  hits,  which  is  also  a  superior  outcome.  Interestingly,  the  “self-trained”  group  had 
the  lowest  performance  during  training,  yet  they  had  greatest  degree  of  transfer.  Aside  from  the 
self-trained  group’s  superior  performance,  the  study  showed  that  groups  that  received  incentives 
performed  better  than  those  that  did  not.  (It  should  be  noted  that  the  participants  pool  were 
university  students — who  are  unlikely  to  have  been  luggage  inspectors — and  the  exercise 
incorporated  very  low  stakes;  a  points  system  that  resulted  in  a  maximum  gain  of  only  $7.00.) 


2.4.5  The  "Common  Eiements"  Theory 

The  initial  meaning  of  "transfer,"  historically  speaking,  is  when  the  learning  of  a  skill  results  in 
improved  learning  of  some  other  skill  that  is  both  different  from  and  yet  similar  to  the  first 
learned  skill  (e.g.,  fewer  trials  to  criterion  on  learning  skill  2  than  on  learning  skill  2).  The 
primary  theory  of  transfer,  mentioned  earlier  in  this  Report,  is  the  "common  elements"  theory 
first  proposed  by  Edwin  Lee  Thorndike: 

One  mental  function  or  activity  improves  others  insofar  as  and 
because  they  are  in  part  identical  with  it,  because  it  contains 
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elements  common  to  them.  Addition  improves  multiplication 
because  multiplication  is  largely  addition...  (1906,  p.  243). 

This  theory  emerged  in  the  context  of  a  theory  that  all  learning  was  the  formation  of  specific 
stimulus-response  habits,  and  research  on  transfer  of  simple  perceptual-motor  skills  (e.g.,  typing, 
pursuit  rotor  tasks).  There  was  concern,  of  course,  with  educational  applications,  such  as  the 
possibility  of  transfer  of  language  skill  (e.g.  the  learning  of  Latin  will  increase  the  student's 
ability  to  learn  French).  The  common  elements  theory  still  serves  as  a  guide  to  research,  though 
the  topics  and  methods  for  study  have  expanded  considerably  since  Thorndike's  pioneering 
studies  (Thorndike  &  Woodworth,  1901).  For  instance,  there  has  been  a  burgeoning  of  research 
on  problem  solving  by  analogical  transfer  (see  Hoffman,  Eskridge  &  Shelley,  2009). 

The  common  elements  theory  leaves  open  the  matter  of  what  constitutes  an  element.  Thus,  we 
can  find  the  theory  manifested  in  computational  models  in  which  the  common  elements  are 
production  rules  that  apply  to  different  kinds  of  problems  (Singley  &  Anderson,  1989).  With  the 
emergence  of  cognitive  engineering  and  studies  of  human-computer  interaction  there  was 
another  extension  of  research  on  transfer,  showing  positive  transfer  from  learning  to  use  one 
software  system  to  learning  another  if  the  two  share  the  functionalities  (e.g.,  of  text  editing)  (see 
Singley  &  Anderson,  1989).  However,  if  the  functionalities  are  similar  but  the  display  and  input- 
output  mechanisms  are  very  different,  there  can  be  little  transfer.  Jahn,  Krems,  and  Gelau  (2009) 
showed  only  slight  positive  transfer  after  100  trials  at  using  one  in-vehicle  route-planning  GPS 
system  to  using  another. 

This  lack  of  transfer  reflects  the  differences  between  the  systems  in  controls,  in  displayed 
information,  and  in  optimal  monitoring  procedures.  The  differences  in  controls  alone  may  have 
been  sufficient  to  prevent  transfer,  as  results  from  laboratory  studies  suggest  (e.g.,  Pashler  & 
Baylis,  1991).  Both  greater  positive  transfer  as  well  as  negative  transfer  might  occur  between 
more  similar  systems  (p.  148). 

What  this  consideration  means  is  that  transfer  cannot  be  analyzed  separately  from  a  notion  of 
contextual  dependency.  This  leads  directly  to  the  notion  of  "transfer  appropriate  processing." 


2.4.6  Transfer-Appropriate  Processing 

Transfer  from  the  training  context  to  the  evaluation  or  performance  context  depends  on  the 
similarity  of  the  tasks  comparing  the  two  contexts  (e.g.,  Adams,  Kasserman,  Perfetto,  Bransford 
&  Franks,  1988;  Morris,  Bransford,  &  Franks,  1977;  Needham  &  Begg,  1991;  Perfetto, 
Bransford  &  Franks,  1983).  In  a  study  by  Healy  et  al.  (2005),  participants  executed  a  time 
estimation  task.  Durability  of  performance  over  long  delays  could  be  achieved  if  the  procedures 
were  exactly  duplicated  from  trial  to  trial,  but  when  the  tasks  were  changed  even  slightly  from 
trial  to  trial,  very  little  transfer  occurred.  Thus,  for  transfer  to  be  effective,  “the  training  must 
incorporate  the  complete  set  of  transfer  task  requirements,  including  any  secondary  task 
requirements  imposed”  (p.ll). 

In  its  initial  incarnation,  the  finding  of  transfer-appropriate  processing  was  used  to  argue  against 
tenets  of  levels-of-processing  theory  (Craik  &  Lockhart,  1972)  and  the  supposed  encoding 
strength  of  deeper  processing  (Morris,  Bransford,  &  Franks,  1977).  Some  had  argued  that,  rather 
than  the  depth  of  processing,  “it  is  the  qualitative  nature  of  the  task,  the  kind  of  operations 
carried  out  on  the  items  that  determines  retention”  (Craik  &  Tulving,  1975).  Original 
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investigations  of  this  theory  focused  on  recognition  and  recall,  but  it  has  been  greatly  expanded 
upon  to  account  for  dissociations  in  implicit  and  explicit  memory  tasks  for  verbal  and  pictorial 
stimuli  (e.g.,  Roediger  &  Blaxton;  1987;  Roediger,  Weldon,  &  Challis  1989). 

Transfer  Appropriate  Processing  theory  has  additionally  encompassed  more  complex  cognitive 
processes.  For  example,  within  problem  solving  research,  TAP  theory  supports  the  notion  that 
initial  strategies  influence  later  problem  solving  and  that  the  matching  of  strategies  during 
learning  and  test  facilitates  overall  problem  solving  effectiveness.  This  research  has  been 
conducted  on  simple  puzzle  tasks  (Adams,  et  al.  1988)  but  also  on  more  complex  tasks  such  as 
learning  to  use  graphics  software  (Caplan  &  Schooler,  1992).  Finally,  TAP  theory  has  been 
integrated  into  the  tenets  of  Problem-Based  Learning  (Hmelo,  1998). 

More  recently,  TAP  theory  was  used  to  account  for  varieties  of  findings  in  category  learning. 
Markman  and  Ross  (2003)  suggest  that: 

...  category  acquisition  occurs  in  the  course  of  using  categories  for 
different  functions.  The  particular  information  that  is  acquired  about 
a  category  member  in  the  context  of  carrying  out  a  particular  task 
depends  on  the  information  that  is  required  to  carry  out  that  task 
successfully...  categories  are  critical  for  a  wide  variety  of  tasks,  and 
one  would  hope  that  category-learning  research  can  provide  some 
further  ideas  of  category  learning  and  representation  to  these  areas 
in  which  category  use  is  so  important  (pp.  609-610). 

From  the  perspective  of  understanding  proficiency  and  its  acquisition,  what  is  important  to 
recognize  with  TAP  theory  is  that  synchrony  between  processes  engaged  during  the  time  of 
learning  or  acquisition  of  a  given  material  and  the  eventual  use  of  that  material  is  crucial  for 
performance  across  a  surprising  number  of  tasks  (Roediger,  et  al.  2002).  Contextual  factors, 
therefore,  are  critical  to  learning  and  retention  over  and  above  what  is  typically  described  in  the 
memory  literature.  More  specifically,  TAP  theorizing  is  most  cogent  with  respect  to  acceleration 
in  that  TAP  theory  has  consistently  identified  that  recapitulating  specific  encoding  and  retrieval 
operations  enhances  performance. 

This  leads  directly  to  the  question  of  training  for  transfer  in  domains  of  high  complexity.  The 
implication  is  that  ecological  validity  of  the  training  content  and  training  processes  will  permit 
transfer,  in  the  sense  of  performance  carry-over  from  the  training  to  the  operational  context. 
Fidelity  per  se  is  not  the  critical  feature.  A  substantial  body  of  research  suggests  that  only  some 
aspects  of  the  training  need  to  be  faithful  to  the  operational  setting.  For  example,  research  has 
found  that  use  of  high  physical  fidelity  simulations  in  training,  had  very  little  effect  on  actual  job 
tasks  compared  with  lower-fidelity  simulations  (Taylor  et  al.,  1993).  Similarly,  research  has 
successfully  used  low  fidelity  PC  based  simulations  to  train  complex  individual  and  teamwork 
skills  (Gopher,  Weil,  &  Bareket,  1994;  Jentsch  &  Bowers,  2001;  Prince  &  Jentsch,  2001;  Taylor 
et  al.,  1999).  The  level  of  fidelity  needed  should  be  driven  by  the  processes  and  activities  in  the 
task  (Salas,  Bowers,  &  Rhodenizer,  1998;  Salas  &  Burke,  2002).  This  idea  has  led  training 
research  to  consider  the  relevance  of  cognitive  fidelity  in  today’s  synthetic  task  environments 
(see  Durlach,  &  Mavor,  1995;  Entin,  Serfaty,  Elliott,  &  Schiflett,  2001). 
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2.4.7  Within-domain  Transfer  Across  Problem  Types 

A  focus  of  research  in  Expertise  Studies  is  the  acquisition  of  skill  in  domains  where  problems  are 
complex,  often  ill- structured  and  highly  reliant  on  knowledge.  The  transfer  question  is  whether 
knowledge  particular  to  some  sub-domain  can  be  utilized  in  solving  problems  in  some  other  sub- 
domain.  While  this  echoes  the  Thomdikian  definition  of  transfer,  as  between  two  skills  that  are 
both  similar  and  different,  the  focus  here  is  on  knowledge.  Feltovich,  et  al.  (1992)  investigated 
the  role  of  complex  knowledge  in  medical  clinics.  The  hypothesis  from  common  elements  theory 
would  be  that: 


Transfer  (and  knowledge  application)  requires  that  a  common 
structure  between  different  contexts  be  detected,  imposed,  or 
assembled,  so  that  the  applicability  of  a  pertinent  body  of 
knowledge  in  the  different  (or  perhaps  just  differently  appearing) 
circumstances  can  be  signaled  (p.  214). 

It  is  their  contention  that  achieving  the  ability  to  derive  a  common  structure  across  varying 
context  becomes  more  difficult  as  tasks  become  more  complex  and  as  the  structure  of  the  task 
becomes  increasingly  irregular,  ill-defined,  or  less  orderly.  Ill-structured  domains  are  those  in 
which  multiple,  interacting  concepts  may  apply  to  the  same  case,  and  the  same  concept  may 
apply  across  a  variety  of  cases.  The  learner  must  be  able  to  detect  the  common  structure  of 
concept  across  cases  and  be  able  to  understand  the  ways  in  which  the  context  changes  the  nature 
of  the  concept. 

If  the  findings  in  Expertise  Studies  and  the  tenets  of  Cognitive  Flexibility  Theory  are  true — that 
experts  are  able  to  adapt  to  novel  cases — then  the  transfer  of  capability  across  different-yet- 
similar  problem  types  across  subdomains  but  within  a  single  within  a  domain  is  a  necessary 
aspect  of  high  proficiency.  Fundamental  to  the  achievement  of  robustness,  resilience  and 
adaptivity  is  the  opportunity  to  practice  at  problems  that  stretch  current  competency  (Feltovich, 
et  al.,  1997).  Professionals  must  acquire  knowledge  and  reasoning  skills  that  pertain  to  critical 
domain  goals  but  which  must  be  exercised  in  widely  differing  situations  or  contexts. 

Current  challenges  for  military  training  involve  two  different  sorts  of  transfer.  One  is  transfer 
across  mission  types.  An  example  would  be  an  infantry  commander,  who  knows  traditional 
warfare,  who  is  asked  to  develop  tactics  for  an  insurgency  operation.  The  second  challenge  is 
transfer  across  responsibilities.  An  example  would  be  a  warfighter  having  a  skill  at  maintenance 
of  an  F-16  engine  who  is  promoted  to  a  supervisory  position;  another  example  is  the  focus  on 
every  Soldier  or  Airman  becoming  a  “sensor,”  requiring  the  infantryman  to  understand  the 
rudiments  of  Intelligence  data  collection.  Since  different  skill  sets  would  be  involved,  one  would 
traditionally  need  to  train  for  the  new  role,  and  not  just  assume  transfer  would  or  might  happen. 

In  their  paper  on  the  need  for  "adaptive  expertise"  in  the  changing  workplace.  Smith,  Ford  and 
Kozlowski  (1997)  argued  that  little  hope  should  be  held  out  for  transfer  of  general  knowledge  to 
specific  tasks  or  skills  in  complex  cognitive  work,  but  hold  out  hope  for  training  and  transfer 
benefits  from  new  methods  such  as  advance  organizers,  discovery-based  learning,  error-based 
learning,  varied  practice,  and  cognitive  mapping  exercises  to  build  mental  models. 
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2.4.8  Generalizations  About  Transfer 

As  the  above  discussion  shows,  it  is  difficult  to  latch  upon  generalizations  concerning  transfer. 
Thorndike’s  initial  studies  of  learning  and  transfer  were  based  on  the  “doctrine  of  formal 
discipline”  that  was  entailed  by  the  associationist  view  that  learning  is  the  strengthening  of 
stimulus-response  bonds  (a  version  of  Aristotle's  law  of  frequency).  Studies  did  not  clearly 
support  the  prediction  this  theory  makes  for  transfer.  For  instance,  one  of  the  early  studies 
involved  training  students  to  estimate  the  length  of  lines  less  than  2  inches  long  and  then  transfer 
to  estimating  the  length  of  lines  grater  than  6  inches.  There  was  no  positive  transfer  apparent  in 
the  data  (Mayer,  2003).  At  higher  levels  of  learning,  no  transfer  was  found,  for  instance,  from  the 
learning  of  Latin  to  learning  in  other  languages  at  the  high  school  level.  On  the  other  hand, 
learning  to  read  a  passage  while  crossing  out  tertian  letters  lent  positive  transfer  in  reading 
passages  while  crossing  out  other  letters  (Thorndike  &  Woodworth,  1901).  Ironically, 
Thorndike’s  initial  doctrine  of  formal  discipline  motivated  the  “Latin  school  movement”  which 
argued  that  learning  Latin  promotes  the  learning  of  English,  and  learning  in  general.  Transfer  is 
specific,  and  not  general. 

Pillsbury  reviewed  the  research  on  transfer  in  1926,  with  regard  to  the  educational  doctrine  of 
formal  discipline,  for  example,  that  training  memory  for  one  task  context  will  train  memory  for 
all  other  task  contexts.  He  cited  an  extensive  study  in  which  university  students  were  tested  for 
skill  in  memorixing  poetry  and  then  given  extensive  practice  at  memorizing  nonsense  material, 
after  which  they  took  another  poetry  memorization  test.  He  found  only  modest  evidence  for 
improvement.  Subsequent  experiments,  with  control  conditions,  failed  to  replicate  this  result  and 
attributed  better  performance  on  the  second  test  to  simply  having  had  taken  the  initial  test. 
Interestingly,  Pillsbury  advocated  that  learning  should  be  by  "ideas"  rather  than  by  "rote,"  a 
principle  echoed  in  essentialy  all  modem  theories  of  meaningful  or  active  learning,  and  in 
educational  resarch  as  well  (e.f..  Cognitive  Flexibility  Theory).  Pillsbury  notes  that  any  positive 
transfer  might  not  necessarily  be  due  to  the  training  of  a  common  skill  or  "faculty,"  but  might 
instead  be  due  to  the  adoption  of  some  general  rule  or  habit  (or  even  something  as  simple  as 
learning  to  be  more  careful  when  in  the  psychology  laboratory!).  On  the  other  hand,  some 
remnant  of  common  elements  theory  is  preserved  in  the  findings  Pillsbury  summarizes.  Learning 
Latin  can  lead  to  modest  improvement  in  skill  in  English.  This  is  commonly  explained  because 
of  similarities  of  grammatical  and  syntactical  rules,  but  Pillsbury  argued  that  Eatin  is  heavily 
case  sensitive  versus  syntax-denendent,  and  so  any  transfer  is  almost  certainly  due  to  the  sharing 
of  words  by  word  origins.  To  paraphrase  Pillsbury  (pp.  308-313): 

Biology  training  improves  skill  at  observing  living  materials; 
studying  descriptive  geometry  improves  skill  at  solving  geometry 
problems;  training  in  special  skills  does  not  improve  overall 
performance  in  school.  Overall,  the  major  variable  that  stood  out  in 
the  resarch  findings,  even  in  the  early  1900s,  was  general  level  of 
intellectual  ability.  While  it  is  desirable  to  get  training  in  special 
fields,  the  student  should  be  prepared  to  find  very  little  application 
for  much  of  his  knowledge. 

One  resolution  to  the  mystery  is  to  recognize  that  there  are  complex  interactions.  For  instance, 
there  is  a  so-called  "paradox"  of  transfer.  Certain  training  practices,  such  as  presenting  learners 
with  an  increased  amount  of  information,  or  increasing  the  pace  of  training,  can  enhance  learning 
during  training  but  may  not  promote  high  levels  of  performance  subsequently  in  the  operational 
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context  (the  "durability"  meaning  of  transfer)  (Schneider,  Healy  &  Bourne,  2002).  Practices  such 
as  increasing  task  variability  will  interfere  with  performance  during  training,  but  enhance 
transfer  (Wulf  &  Schmidt,  1988).  Reasons  for  these  phenomena  include  a  distinction  between 
learning  that  allows  for  immediate  but  temporary  recall,  and  learning  that  allows  for  longer-term 
recall  that  is  procedural  or  skill-like  rather  than  purely  declarative  of  knowledge-like. 

In  concluding  her  summary  of  research  pertinent  to  transfer,  Healy  (2007;  Healy,  et  ah,  2003) 
argues  that  transferring  the  skill  to  new  situations  is  often  difficult  but  can  be  promoted  by 
following  a  number  of  training  principles:  Employing  deliberate  practice,  increasing  the 
variability  of  practice,  adding  sources  of  contextual  interference,  using  a  mixed  practice 
schedule,  distributing  practice  in  time,  and  providing  process  and  outcome  feedback  in  an 
explicit  analysis  of  errors. 


2.5  Decay  and  Retention 

The  challenge  of  achieving  and  maintaining  high  proficiency  is  compounded  in  the  military  by 
such  practices  as  collateral  assignment,  redeployment  (e.g.,  rapid  skill  decay  on  the  part  of  pilots 
creates  a  need  for  expensive  re-training),  inadequate  or  ad  hoc  mentoring,  and  the  drive  for  “just- 
in-time”  training. 

2.5. 1  Introduction 

Decay  and  retention  of  both  perceptual-motor  skills  and  declarative  knowledge  have  been 
extensively  studied  by  psychologists  using  methods  including  recall  and  recognition.  Indeed,  the 
experimental  study  of  learning-decay  curves  marks  the  beginning  of  the  experimental 
psychology  of  learning  (Ebbinghaus,  1885;  Woodworth,  1903,  1938).  Also  in  the  learning 
literature,  memory  decay  is  one  of  the  main  hypothetical  causes  of  forgetting  (e.g.,  Anderson, 
2000;  Bransford,  1979).  Particularly  pertinent  to  the  present  study  is  research  on  retention  of 
knowledge  or  skills  following  periods  of  non-use,  which  we  call  "hiatus." 

Retention  of  skill  is  an  issue  in  many  domains  of  professional  activity.  Eor  example,  special 
efforts  and  procedures  are  needed  to  help  physicians  maintain  skill  and  continuously  update  their 
knowledge  (Davis,  2009).  Medicine  contains  formal,  mandated  procedures  for  the  conduct  of 
continuing  medical  education,  including  such  things  as  workshops  and  seminars,  but  also 
periodic  processes  of  recertification.  In  parallel,  there  are  efforts  to  research  and  document  of 
"best-evidence"  practice  in  medicine,  dealing  with  diagnosis,  procedures,  treatment,  and 
management.  Continual  learning  must  be  aimed  at  preventing  medical  error  related  to  the  lack  of 
practice.  These  studies  are  conducted  by  various  bodies,  including  venues  of  traditional 
scholarship  and  major  national  studies  (e.g.,  by  the  Institute  of  Medicine).  They  present 
prescriptions  for  how  to  support  continuing  medical  education. 

As  the  technology  and  capability  of  military  operations  have  increased,  so  too  has  there  been  an 
increase  in  the  number  of  jobs/roles  that  are  complex  and  knowledge  intensive,  significantly 
raising  the  baseline  or  benchmark  for  minimal  proficiency.  Indeed,  it  is  often  said  that  in 
complex  sociotechnical  work  systems  (of  all  kinds)  expertise  is  a  must  because  decisions  are 
high  stakes  and  high  risk.  Thousands  of  military  jobs  can  still  be  defined  largely  in  terms  of 
"common  tasks"  and  associated  procedures,  but  thousands  can  only  be  defined  in  terms  of  the 
need  for  significant  levels  of  knowledge  and  significant  skill.  But  of  those  thousands  that  are  still 
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largely  definable  in  terms  of  tasks,  they  too  are  morphing,  as  is  witnessed  in  the  diverse  set  of 
activities  that  are  new  to  the  traditional  infantry.  The  very  concept  of  "task"  has  been  questioned 
for  its  appropriateness.  It  is  no  longer  the  case  that  jobs  can  be  defined  in  terms  of  "tasks" 
defined  as  fixed  sequences  of  activities.  Workers  engage  in  context-sensitive,  knowledge-driven 
choice  among  activities  (Rasmussen,  1986,  1992).  In  other  words,  there  is  a  general  shift  from 
simple  skills  to  high-level  cognitive  skill  sets. 

Given  this,  one  should  be  cautious  about  generalizing  from  the  research  on  skill  decay/retention, 
since  the  vast  majority  of  the  research  involves  studies  of  single  and  relatively  simple  skills,  the 
prototypical  ones  being  simple  motor  tasks,  tracking  tasks,  and  vocabulary  learning  tasks. 

Psychological  explanations  of  memory  and  skill  loss  invoke  concepts  such  as  decay  and 
interference,  memory  trace  strength,  retrieval  blockage,  and  "dilution"  of  memories  (see 
Landsdale  &  Baguley,  2008;  Wixted,  2004).  All  the  models  attempt  to  explain  how  there  is  a 
proportional  decline  in  memory  loss  as  a  function  of  the  age  of  the  memory  (i.e.,  Jost's  Law  that 
of  two  memory  traces  of  equal  strength  at  a  given  time,  the  younger  trace  will  decay  more 
rapidly  than  will  the  older). 

Almost  all  the  research  is  academic,  and  looks  at  tasks  involving  memory  of  simple  materials 
such  as  words  word  pairs,  or  sentence  or  pictures.  This  leaves  a  rather  large  conceptual  gap 
between  this  research  base  and  issues  of  retention  of  knowledge  and  skill  necessary  for  the 
conduct  of  complex  jobs. 


2.5.2  General  Findings  about  Decay  and  Decay  Rates 

Retention  of  skill  can  be  evaluated  in  a  number  of  ways.  One  is  to  compare  performance  at  the 
end  of  a  training  period  with  performance  after  a  retention  interval  during  which  there  was  no 
practice.  This  evaluation  reflects  performance  loss.  Another  method  is  to  look  at  the  obverse, 
using  the  "method  of  savings":  it  usually  takes  fewer  trials  or  less  time  to  re-achieve  some 
criterion  level  of  performance  than  it  took  originally  to  achieve  it.  This  is  taken  to  reflect  the 
strength  of  the  original  learning. 

The  Extent  of  Decay 

Rubin  and  Wenzel  (1996)  analyzed  data  from  over  200  published  studies  that  demonstrated  the 
decay  of  long-term  memory.  With  the  exception  of  autobiographical  memory  (which  has  certain 
unique  aspects),  other  materials  show  the  typical  decay  curve  in  which  most  forgetting  occurs 
within  some  brief  span  immediately  after  the  original  learning,  typically  within  the  first  year  up 
to  about  three  years.  (While  most  of  the  data  sets  were  from  studies  of  declarative  memory,  not 
all  were.  For  instance,  one  study  was  of  the  memory  of  infants  who  had  learned  to  kick  at  the 
sight  of  a  crib  mobile.) 

Arthur,  et  al.  (1998)  reviewed  the  literature  on  skill  decay,  incorporating  summary  conclusions 
from  a  number  of  earlier  review  articles. 

The  effect  of  progressive  knowledge  or  skill  deterioration  when 
knowledge  and  skills  are  not  used  or  exercised  for  extended 
periods  of  time  is  a  fairly  robust  phenomenon.  Although  the  vast 
majority  of  the  literature  consists  of  laboratory  studies,  in  applied 
settings,  this  is  related  to  infrequent  opportunities  to  practice  or 
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perform  acquired  skills  (p.  59). 

In  general,  the  relation  between  skill  retention  and  the  length  of  the  non-practice  or  nonuse 
interval  is  negative.  Arthur,  et  ah,  (1998)  derived  a  broad  generalization:  After  a  year  of  non-use 
or  non-practice,  the  average  person  is  performing  at  less  than  92%  of  their  performance  level 
before  the  beginning  of  the  non-practice  interval. 

The  Shape  of  the  Decay  Curve 

Studies  of  loss  of  knowledge  that  was  acquired  in  academic  courses  consistently  show  that 
considerable  forgetting  typically  occurs  within  the  first  few  years  following  instruction  (e.g.. 
Bacon  &  Stewart,  2006;  Semb  &  Ellis,  1994).  This  same  general  finding  of  initial  rapid 
forgetting  followed  by  negatively-accelerated  forgetting  has  obtained  in  some  studies  of  motor 
skill  learning  as  well  as  cognitive  learning,  although  motor  skill  generally  shows  less  decay  than 
cognitive  learning  (Arthur,  et  ah,  1993). 

The  rate  of  decay  does  not  depend  on  the  degree  of  learning — that  is,  although  the  amount  of 
original  learning  may  vary,  the  rate  of  forgetting  is  similar.  In  an  experiment  for  the  military, 
Schendel  and  Hagman  (1991)  found  that  the  rate  of  decay  did  not  differ  even  though  initial 
performance  for  high  and  low  ability  soldiers  differed  (see  also  Wisher,  1999). 

Figure  2.1  shows  bombing  accuracy  data  reported  from  an  actual  training  environment  for  fighter 
pilots  coming  off  of  14  hours  of  training  at  Fallon  Naval  Air  Station.  This  figure  can  be  taken  as 
representative  of  leaming/forgetting  curves,  showing  substantial  levels  of  decline  after  only 
several  weeks. 


Bombing  Accuracy  of  F/A-18  Pilots 


Air-to-Ground  Weeks  After  Leaving  NAS  Falloa 

training  at  NAS 
Fallon.  U.S.  Navy 
“Strike  University” 

(total  14  flyiiig  hours) 


Figure  2.1.  A  performance  decay  (or  “forgetting”)  curve. 
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The  “decay”  component  of  such  performance  curves  is  idealized  in  Figure  2.2 


Performance 


Time  without  Practice  or  Rehearsal 


Figure  2.2.  An  Idealized  performance  decay  curve. 


Harry  Bahrick  and  his  colleagues  conducted  a  classic  series  of  studies  on  long-term 
retention/decay  for  mathematics  and  for  Spanish  learned  in  high  school,  on  the  part  of  literally 
thousands  of  people,  tracking  retention  in  some  cases  out  as  long  as  50  years  (Bahrick,  1979, 
1983,  1984;  Bahrick,  Bahrick,  &  Wittlinger,  1975;  Bahrick  &  Phelps,  1987).  These  studies  found 
rapid  and  significant  levels  of  forgetting  in  the  few  years  after  students  had  taken  a  one-year 
course  in  Spanish.  Forgetting  then  levels  off,  with  some  knowledge  being  remembered  after  long 
periods  of  nonuse.  According  to  Bahrick,  successive  relearning  is  essential  for  the  retention  in 
the  memory  "permastore"  (1984).  (In  the  case  of  memory  for  a  second  language,  this  would  no 
doubt  entail  use,  and  not  just  relearning.) 

Research  consistently  finds  that  the  performance  decline  is  rapid  just  following  the  training,  but 
that  the  rate  of  decline  decreases  (i.e.,  negatively  accelerated)  (Arthur  et  al,  1998;  Driskell  et  al. 
1992;  Wixted  &  Ebbesen,  1991;  see  also  Lance  et  al.,  1998).  Figure  2.2  illustrates  this  concept. 
During  training,  proficiency  increases  to  some  level  but  decreases  over  time  after  training.  This 
can  be  modeled  using  a  number  of  two-parameter  equations,  such  as  Anderson  and  Schooler's 
(1991)  power  law  of  forgetting  based  on  assumptions  about  the  decay  of  memories  (i.e.,  the 
strength  of  a  given  memory  is  a  log  function  of  original  memory  strength  and  log  time  since 
learning)  (see  also  Wixted  &  Ebbesen,  1997).  Generally,  forgetting  curves  suggest  that  humans 
lose  nearly  50%  of  their  memory  for  new  material  (in  declarative  knowledge  learning  tasks) 
within  a  day  unless  some  form  of  review  is  engaged  (see  Wixted,  2004  for  a  discussion). 

The  literature  on  the  rate  of  skill  decay  for  cognitively-based  tasks  (i.e.,  problem  solving  rather 
than  perceptual-motor  skills)  seems  to  suggest  similar  rates  of  decay.  Wetzel  et  al.  (1983)  looked 
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at  learning  and  retention  for  a  perceptual-cognitive  task  (oceanographic  analysis)  and  found  21% 
decrease  in  scores  after  a  month  (Wetzel,  Konoske  &  Montague,  1983)  and  problem  solving 
skills  were  shown  to  drop  16%  after  two  months  (Austin  &  Gilbert,  1973). 

The  Decay  Curve  in  Military  Jobs/Tasks 

Numerous  studies  of  retention  in  military  domains  have  demonstrated  decay.  For  instance, 
Wetzel,  et  al.  (1983)  followed  20  individuals  who  were  receiving  training  in  antisubmarine 
warfare,  comparing  performance  during  an  acoustic  analysis  training  course  and  performance 
after  25-days  of  "non-utilization,"  to  determine  what  they  had  retained  from  the  course  after  this 
period.  Both  pre  and  post  conditions  used  two  kinds  of  tests,  multiple  choice  and  performance 
measures.  The  learners'  knowledge  and  skill  were  found  to  "degrade  significantly." 

The  problem  of  skill  decay  among  military  personnel  as  a  consequence  of  rotations  or  hiatus 
periods  has  been  documented  for  some  time  (e.g..  Wisher,  et  al.,  1991)  and  not  just  in  the  U.  S. 
Military  (see  Henik,  et  al.,  2001). 

Studies  of  the  retention  of  soldier  tasks  have  shown  effects  of  degree  of  initial  learning,  active 
practice  (practice  on  actual  equipment  rather  than  in  the  schoolhouse),  and  spaced  practice 
(Stothard  &  Nicholson,  2001).  In  an  extensive  and  well-controlled  study,  Kneer,  et  al.  (1984) 
evaluated  soldier  retention  of  eight  procedures  in  the  repair  of  armored  equipment.  Their  study 
confirmed  the  impact  of  the  above  variables.  The  procedural  tasks  that  were  examined  varied  in 
length,  complexity,  and  amount  of  training/practice  during  actual  operations.  Performance  was 
measured  after  training,  then  four  weeks  later,  and  in  some  cases  up  to  72  months  after  training. 
The  results  support  other  retention  studies  that  show  rapid  decay  after  training,  with  fairly  stable 
performance  over  longer  periods  of  time. 

Henik,  et  al  (2001)  examined  skill  decay  on  the  part  of  tank  gunners  and  missile  aimers  (about  80 
of  each)  who  had  just  been  released  from  military  service.  Six  and  eighteen  months  later,  their 
skill  was  tested  in  multiple  scenarios,  and  their  declarative  memory  was  tested  as  well.  Findings 
showed  very  small  but  statistically  significant  loss  of  declarative  knowledge  about  the  job  (about 
a  5%  drop),  most  of  this  occurring  between  12  and  18  months.  The  findings  also  showed  a  more 
dramatic  loss  of  skill  (decreased  average  and  increased  variability  in  speed  and  accuracy). 

Marmie  and  Healy  (1995)  studied  the  retention  of  skill  at  tank  gunnery,  a  job  requiring  both 
discrete  and  continuous  skills,  both  motor  skills  and  cognitive  skills.  After  12  trainings  sessions 
over  four  weeks,  there  were  retention  tests  from  one  out  to  22  months.  The  main  finding  was  the 
typical  decay  curve,  that  is,  rapid  initial  loss  of  skill. 

Lance,  et  al.  (1998)  investigated  skill  decay  for  a  number  of  U.  S.  Air  Force  enlisted  specialties 
(including  air  traffic  controller,  ground  equipment  mechanic,  radio  operator,  avionics,  others.). 
An  attempt  was  made  to  link  decay  to  such  variables  as  aptitude  (data  from  personnel  files),  task 
difficulty  (ratings  data  from  the  USAF  task  inventory),  initial  skill  level  (gauged  on  the  basis  of 
participant  experience),  and  performance  level  in  "last  time  performed"  reports.  The  results 
showed  the  expected  relation  of  performance  to  retention  interval  (and  fairly  high  retention  for 
the  less  complex  tasks).  This  and  other  studies  permit  the  extension  of  the  general  skill/memory 
decay  finding  from  academic  research  (i.e.,  of  simple  perceptual-motor  tasks  and  declarative 
memory  tasks)  to  military  jobs. 

However  the  relation  of  retention  interval  to  amount  of  decay  was  not  strong  since  other 
moderator  variables  were  at  work.  With  regard  to  those  moderator  variables,  there  were  many 
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interactions  and  no  strong  correlations.  As  in  all  research  in  this  area,  there  are  many 
complicating  considerations  of  experimental  design  and  control  for  potential  confounds.  For 
example,  studies  need  to  control  for  initial  over-learning.  As  another  example,  during  a  hiatus  an 
individual  may  not  have  conducted  their  primary  tasks  but  might  conduct  similar  ones. 
"Additional  unmeasured  performance  determinants  (e.g.,  motivation,  prior  job  knowledge)  may 
have  overshadowed  the  effects  of  extended  retention  intervals  on  task  performance"  (p.  118). 

Factors  that  Influence  Decay 

Major  factors  that  influence  the  decay  or  retention  of  trained  skills  over  extended  periods  of 
nonuse  are  (a)  the  length  of  the  retention  interval,  (b)  the  degree  of  overlearning,  (c)  certain  task 
characteristics  (e.g.,  closed-looped  vs.  open-looped  tasks,  physical  vs.  cognitive  tasks),  (d) 
methods  of  testing  for  original  learning  and  retention,  (e)  conditions  of  retrieval,  (f)  instructional 
strategies  or  training  methods,  and  (g)  individual  differences. 

Arthur,  et  al  (1998)  drew  a  valuable  distinction  between  methodological  and  task  factors. 
Methodological  factors  are  those  that  can  be  modified  in  the  training  or  learning  context  to 
reduce  skill  loss  (e.g.,  degree  of  over-learning,  conditions  of  retrieval). 

Task  Factors 

Task  factors  are  inherent  characteristics  of  the  task  and  are  typically  not  amenable  to 
modification  (e.g.,  motor  versus  cognitive  tasks). 

Schendel,  Shields,  and  Katz  (1978)  reviewed  the  research  on  retention  of  motor  skills  and 
concluded  that  the  most  important  determiner  of  retention  is  the  level  of  performance  achieved 
prior  to  hiatus.  Arthur,  et  al.,  conclude  that  the  single  most  important  of  these  variables  appears 
to  be  the  amount  or  degree  of  over-learning  as  a  consequence  of  training  beyond  that  required  for 
initial  proficiency.  Over-learning  is  the  deliberate  practice  beyond  the  point  of  having  reached 
some  criterion  level  of  performance  (Driskell,  et  al.,  2992).  The  degree  of  over-leaning  can  be 
scaled  in  terms  of  the  number  of  practice  trials  after  achieving  criterion  to  the  number  taken  to 
achieve  criterion  (expressed  as  a  percent). 

Explanations  include  the  basic  law  of  frequency  of  association,  the  possibility  that  over-learning 
prevents  decay,  the  possibility  that  additional  practice  brings  with  it  additional  feedback,  and  the 
possibility  that  over-learning  enhances  trainee  confidence. 

It  has  generally  been  found  that  spaced  practice  during  training  results  subsequently  in  better 
retention  than  massed  practice,  especially  with  a  long  retention  interval,  or  delay  between  the  end 
of  learning  and  the  beginning  of  testing.  In  fact,  optimal  performance  at  test  seems  to  result  when 
the  interval  between  practice  trials  is  approximately  equal  in  length  to  the  retention  interval.  This 
conclusion  seems  to  hold  even  when  practice  is  distributed  across  sessions  rather  than  across 
trials  within  a  session  (Healy,  2007).  Bahrick  and  Phelps  (1987)  showed  that  when  Spanish 
vocabulary  was  learned  over  a  longer  time  period  (six  to  nine  study  sessions  over  30  days  vs.  six 
to  nine  sessions  back-to-back),  the  retention  of  this  material  8  years  later  was  increased. 
Repeated  exposures  lead  to  greater  learning  (Bacon  &  Stewart,  2006;  Halpern  &  Hakel,  2003), 
and  when  those  exposures  are  spread  over  time  they  appear  to  be  even  more  effective. 

This  finding  has  a  direct  implication  for  retraining  following  hiatus.  Many  ideas  about  skill 
retention  during  hiatus  involve  practice  experiences  that  would  inevitably  be  massed  in  the  sense 
of  concentrated  at  particular  times  within  the  practitioner’s  primary  work  schedule.  The  literature 
on  distributed  practice  would  suggest  that  training  during  hiatus  be  distributed. 
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The  effect  of  over-learning  can  be  hidden  by  the  effect  of  mental  rehearsal  or  imaginary  practice 
during  the  retention  interval  (Farr,  1987).  According  to  the  notion  of  encoding  specificity  (Healy, 
2007;  Tulving,  1983)  and  the  similar  notion  of  "procedural  reinstatement"  (Healy,  et  al.,  1992), 
performance  following  a  retention  interval  will  be  better  if  the  retention  task  (the  way 
information  is  encoded)  is  similar  to  the  acquisition  task  (the  cues  used  to  stimulate  a  memory). 
Thus,  skill  retention  appears  to  be  greater  if  the  retention  measurement  is  conducted  in  a  context 
similar  to  that  of  the  original  learning,  e.g.,  the  similarity  of  the  training  device  to  the  actual  job 
equipment  (Farr,  1987;  Schendel,  Shields,  &  Katz  (1978).  In  a  study  of  tank  gunner  skills 
Marmie  and  Healy  (1995)  found  that  if  the  training  is  sufficient  and  if  the  post-retention  task 
itself  relies  on  cues  that  are  present  during  training,  then  retention  of  skills  can  significant  for  up 
to  22  months. 

The  type  of  task  can  determine  the  effect  of  over-learning  on  skill  retention  (perceptual-motor 
skill  versus  cognitive  task)  (Ryan  &  Simons,  1981).  This  is  discussed  in  more  detail  below,  in 
the  section  on  "differential  decay." 

Methodological  Factors 

Different  retention  measures  can  yield  different  degrees  of  apparent  retention  (Farr,  1987).  Many 
choices  have  to  be  made  in  designing  experiments,  and  one  consequence  is  that  different  studies 
use  different  criteria  (Farr,  1987).  “Proficiency”  or  “mastery”  at  a  skill  has  been  defined  as  “one 
errorless  trial.”  Learning  continues  beyond  that  point  that  accuracy  is  perfect,  and,  more 
importantly,  “accuracy  asymptotes  rapidly  in  many  tasks,  leading  to  a  potentially  false 
conclusion  that  the  material  has  been  mastered. . .  [and]  because  learning  continues  past  error- free 
trials,  overlearning  may  really  just  be  a  higher  level  of  skill  acquisition”  (Arthur,  et  al.,  1998;  p. 
91;  see  also  Regian  &  Schneider,  1990). 

The  primary  individual  difference  that  has  been  studied  is  intellectual  ability.  In  general,  low 
ability  learners  and  high  ability  learners  tend  to  learn  different  things,  learn  different  things 
differently,  and  show  different  patterns  of  decay.  Lower  ability  learners  forget  larger  portions  of 
abstract,  theoretical  material  than  do  higher  ability  individuals.  Higher  ability  individuals  retain 
more  knowledge  or  skill  over  periods  of  nonuse  than  lower  ability  individuals,  but  this  can  be 
because  higher  ability  trainees  acquire  more  knowledge  or  skill  during  training  (Farr,  1987). 


2.5.3  Retention  of  Piloting  Skills 

Following  a  hiatus  it  is  necessary  for  fighter  pilots  to  reacquire  a  feel  for  high-performance 
flight,  even  though  pilots  we  interviewed  say  things  like  Flying  is  like  riding  a  bike.  Our  focus  in 
this  Report  is  on  the  retention  of  the  broader  spectrum  of  flight  skills,  including  mission-relevant 
and  tactical  skills.  Research  on  this  topic  constitutes  an  extensive  literature.  Prophet  (1977) 
provided  an  annotated  bibliography  of  120  articles.  In  this  Report,  we  review  selected  studies 
that  are  representative  of  the  main  and  consistent  findings. 

Childs,  et  al.  (1983)  investigated  retention  of  flight  skills  in  civilian  pilots  at  intervals  of  8,  16, 
and  24  months  after  receiving  certification  from  a  course  of  training.  Significant  skill  loss  was 
found  and  manifested  the  forgetting  curve  described  in  the  psychology  literature.  Groups  who 
had  flying  experience  during  the  retention  interval  fared  better  than  those  with  none,  but  they  still 
showed  significant  decrement.  Skills  that  require  appreciable  coordination  between  cognitive 
skills  and  "control  skills"  were  among  those  most  substantially  degraded.  Writing  in  1983, 
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Childs  et  al.  proposed  that  are  training  should  include  "cognitive  training"  (especially  for 
cognitive  cues),  and  "full  mission  simulation." 

Wright  (1973)  examined  retention  of  flying  skills  after  periods  of  non-flying  and  minimal  flying. 
All  of  the  measures  of  flying  proficiency  were  estimates  produced  by  the  subjects  themselves, 
including  not  only  estimates  of  their  ability  at  the  time  of  measure  (after  the  period  non-use)  but 
also  retrospectively  (e.g.,  back  to  when  they  were  first  certified  after  training).  The  findings  show 
that  pilot  judgments  are  consistent  with  findings  about  performance  decline.  Pilots  know  that 
most  skill  loss  occurs  early,  within  the  first  year,  and  that  skill  loss  slows  marginally  after  that. 

To  relativize  the  above  findings,  which  might  be  regarded  as  typical.  Prophet  (1977)  culled  the 
available  literature  for  data  on  "basic  flight  skills"  after  long  retention  intervals.  He  found 
evidence  that  basic  flying  skills  are  retained  fairly  well  for  extended  periods  of  non-use. 
Nevertheless,  Prophet  (like  Childs,  et  al.  and  others)  proposed  that  proficiency/retraining 
demands  can  be  met  by  simulators  and  other  training  devices.  However,  Prophet  also  asserted 
that  that  not  enough  was  known  (then)  about  the  retention  of  "higher  level  tactical  skills"  or 
strategies.  In  concert  with  Prophet's  findings  (1977),  Wright  (1973)  found  that  instrument  flying 
skills  were  shown  to  deteriorate  more  substantially  than  basic  flying  skills  (visual  flight). 

Refresher  Training  for  F-16  Fighter  Pilots 

There  are  no  “tests  of  expertise”  for  fighter  pilots.  As  an  experienced  fighter  pilot  and  fighter 
pilot  trainer  remarked,  “Highly  proficient  pilots  pass  their  qualifications  with  distinction.  They 
can  think,  decide,  and  act  rapidly.  They  can  recognize  patterns  that  others  cannot.  They  can  act 
by  intuition.  They  feel  good  about  their  strengths  and  are  comfortable  with  their  weaknesses.  ” 
Fighter  pilots  are  also  considered  expert  if  they  have  been  through  the  Weapons  School. 

The  refresher  course  can  last  from  one  to  six  months  depending  on  how  long  the  pilot  has  been 
away  from  the  weapon  system  (or  whether  they  are  moving  to  a  new  weapon  system).  According 
to  an  experienced  fighter  pilot  trainer  in  the  F-18  Transition  Course  (Healy,  2008),  fighter  pilots 
show  different  attitudes  upon  entry  to  the  refresher  course:  "1  have  to  catch  up"  versus  "This  will 
be  easy. "  The  former  tend  to  be  more  expert,  and  typically  are  themselves  experienced  trainers. 
"[They  are]  voracious  readers  and  question  askers;  they  feel  passion  about  the  plan  and  the 
mission.  They  recognize  their  own  limitations  and  lack  of  currency.  They  recognize  that  they 
aren’t  going  to  get  smarter  in  their  new  job"  (Healy,  2008). 

The  fighter  pilots  who  express  the  "this  will  be  easy"  attitude  are  typically  ones  who  typically 
have  mid-level  experience  (i.e.,  as  a  light  lead  but  not  as  a  fighter  pilot  trainer).  They  express  an 
attitude  of  wanting  to  "survive  the  program."  Healy  (2008)  argues  that  these  pilots  are  more 
dangerous  because  they  approach  the  refresher  training  on  the  assumption  that  they  will  "get 
smart"  once  they  get  to  their  new  job.  "They  don’t  know  what  they  don’t  know"  (Healy,  2008). 

Healy  also  observed  that  fighter  pilots  who  use  the  simulator  a  lot  tend  to  be  more  passionate, 
and  likely  want  to  be  weapons  officers. 

Trainer  experience  in  the  F-16  Refresher  Training  is  that  no  one  comes  back  after  a  hiatus  of  any 
duration  (more  than  a  few  months)  at  greater  than  80%  proficiency.  For  long  hiatus,  proficiency 
upon  return  is  estimated  at  40-30%. 

As  described  by  Healy  (2008),  the  progression  of  training  goes  from: 
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(Track  1)  Two  (friendlies)  on  one  (adversary)  start  miles  apart  and  converge.  Track  1  is  for  the 
least  experienced  individuals  (out  of  cockpit  for  at  least  8  years).  There  are  242  academic 
modules,  38  sorties,  20  simulation  sorties. 

(Track  2)  Two  against  two.  Track  2  is  for  individuals  out  3-8  years.  There  are  152  academic 
modules,  17  sorties,  14  simulator  sorties. 

(Track  3)  Four  against  four,  flying  low  and  with  bombs.  Track  3  is  for  individuals  out  less  than  3 
years.  35  training  days  include  165  academic  modules;  11  sorties,  13  simulator  sorties. 

(Track  4)  Capstone  of  1  against  4.  The  fighter  pilot's  job  is  to  not  die  and  to  put  effective 
weapons  against  a  target. 

Where  the  fighter  pilot  starts  in  the  refresher  training  depends  on  the  length  of  their  hiatus.  For 
instance,  individuals  whose  hiatus  was  only  a  few  months  would  start  at  step  3.  Also,  there 
would  be  fewer  sorties  with  more  stuff  packed  into  each  sortie. 

After  the  Step  4  capstone,  the  (re)trainee  goes  into  the  operational  Air  Force,  but  training 
continues  there,  using  the  same  progression  (stages  1,  2,  3)  but  adapted  to  the  local  adversary  and 
nature  of  the  deployment.  The  local  check  out  can  be  10-14  sorties,  one  each  of  different  mission 
sets.  The  fighter  pilot  learns  the  local  flying  environment,  the  diversion  routes,  and  the 
instrument  approaches.  The  weapons  on  an  aircraft  change  its  flight  characteristics,  the  g-force 
stresses  on  weapon  mountings,  etc. 

The  dynamic  complexity  of  the  fighter  pilot’s  job  seems  nearly  overwhelming  to  the  outsider 
(Healy,  2008): 

•  There  are  different  versions  of  the  F-16,  each  with  its  own  unique  engines,  software,  etc. 

•  On  any  one  flight  the  aircraft  will  be  configured  for  one  of  many  possible  mission  types 
(reconnaissance,  strike,  air-to-air,  etc.).  “Each  mission  is  the  soup  of  the  day.  ” 

•  There  are  always  new  tactics  to  learn,  and  new  terms  for  planning  and  communication. 
There  are  five  major  tactics  manuals  and  these  are  rewritten  every  2  years. 

•  There  are  new  challenges  of  joint  and  coalition  operations  (e.g.,  the  confusion  over 
whether  “abort"  means  a  plane  moves  off  versus  entire  mission  is  called  off.) 

•  There  are  challenges  to  memory  and  multitasking.  After  a  mission  debrief,  the  pilot  plans 
the  next  day’s  mission.  After  a  training  sortie  and  debrief,  trainees  will  study  for  a 
training  sortie  they  will  do  one  or  two  weeks  in  the  future.  “This  is  a  constant  leapfrog.  ” 

•  The  aircraft  are  always  being  changed.  Manuals  are  rewritten  regularly.  The  plane’s 
software  upgrades  every  two  years.  There  are  upgrades  to  all  the  avionics  and  hardware. 

All  this  being  said,  the  technology  for  human-system  integration  is  improving.  “New  helmet 
means  we  can  use  the  buttons  to  do  more  than  one  thing.  Press,  press  and  hold,  press  twice.  New 
data  display  tells  what  other  aircraft  are  doing.  ” 

Related  to  the  theme  that  deployment  itself  contributes  to  the  achievement  of  proficiency,  in  the 
fighter  pilot  domain  a  great  deal  of  learning  is  accomplished  and  shared  in  the  debriefings.  A 
debriefing  and  group  discussion  can  last  longer  than  the  mission.  If  a  sortie  went  bad,  a 
debriefing  can  last  five  hours. 
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Although  all  debriefings  have  unique  aspects,  there  is  a  schema.  The  sortie  Lead  stands  up,  asks 
questions  that  can  only  be  answered  yes  or  no.  The  Lead  might  already  know  what  the  trainee 
slipped  up.  “Were  you  too  fast  or  slow?”  Questions  are  also  about  tactical  administration.  “Did 
we  follow  the  safety  rules?”  The  trainee  will  recreate  the  mission  on  whiteboard,  using  video 
records  and  instrumentation  data.  Error  points  will  be  highlighted.  A  main  purpose  of  the  debrief 
is  to  identify  root  causes  (e.g.,  for  why  someone  "died"  (e.g.,  the  radar  was  not  looking  in  the 
right  part  of  the  sky).  It  takes  time  to  build  an  accurate  picture  integrating  visual  observations, 
radar  data,  communications  data,  etc.  Then  the  Lead  will  provide  an  instructional  fix  based  on 
the  learning  points.  This  involves  linking  the  debrief  results  with  what  is  programmed  for  the 
next  simulator  sortie. 

The  F-16  training  program  also  includes  training  of  the  trainers,  having  its  own  syllabus. 
Training  sortie  debriefs  include  de-brief  of  the  trainers  with  regard  to  the  training  itself. 


Interviews  with  Fighter  Pilots  concerning  Hiatus  Experiences.  Effects  and  Retraining 

In  May  2009  interviews  were  conducted  with  ten  current  and  retired  USAF  pilots.  Four  were 
retired  F016  fighter  pilots  who  had  achieved  the  rank  of  Eieutenant  Colonel.  Another  had  served 
in  Special  Operations  as  a  helicopter  pilot  in  border  patrol  and  had  achieved  the  rank  of  Colonel. 
Another  had  been  primarily  a  B-52  pilot  then  a  B-2  navigator.  Another  was  primarily  a  UAV 
sensor  operator.  Two  had  experience  flying  AW ACS.  One  had  additional  experience  flying  Joint 
Stars.  Specific  career  data  available  on  three  of  the  interviewees  specified  the  range  of  flight 
hours  logged  between  approximately  1600  and  2340.  All  interviewees  had  experienced  rotation 
out  to  a  staff  job,  which  for  three  interviewees  involved  jobs  close  to  aviation  (e.g..  Flight 
Instructor,  Ground  Controller,  AOC  duty)  or  relatively  distanced  from  it  (e.g..  Course  Director  in 
Military  Studies,  NATO  C2,  RAF  liaison),  followed  by  reentry  into  a  position  which  might  have 
been  more,  or  less  related  to  their  original  flying  assignments  (e.g.,  weapons  school  instructor). 
The  focus  of  the  interviews  was  to  investigate  these  experiences,  especially  regarding  decline  of 
skills  while  on  staff  rotation  and  recovery  of  skill  upon  rotating  out  of  the  staff  position,  back  to 
a  more  front-line  position. 

There  was  general  agreement  that  skills  do  decline  during  staff  rotations.  One  interviewee 
asserted  that  "Most  of  my  deployments  I  have  never  seen  skill  decay.  Even  holding  12  hours  over 
Iraq,  even  in  a  complex  sorties,  and  not  doing  air-to-air  skills,  you  do  not  get  skill  decay."  But 
then  he  added  "Two-three  weeks  do  some  fancy  flying,  and  you  are  ready  quickly." 

There  was  general  agreement  that  loss  of  basic  flying  skills  is  less  an  issue  than  loss  of  tactical 
skills,  and  that  practice  during  hiatus  at  flying  a  simulator  would  by  itself  not  be  sufficient: 
"Good  sims  at  the  staff  job  aren't  enough.  You  need  to  have  current  guys  to  fly  with".  On  the 
other  hand,  there  was  also  a  consensus  that  it  might  not  take  much  in  the  way  of 
maintenance/refresher  training  to  maintain  tactical  skill: 

"Two  sorties  per  month,  I  could  probably  keep  him  tactically.  Even 
flying  in  the  back  seat  would  not  hurt.  Monthly  basis  is  needed  to 
keep  an  edge...  A  reserve  guy  said,  we  have  all  these  reservists  flying 
5-8  sorties  a  month,  keeps  them  able  to  take  off  and  land  and  fly.  If  I 
saved  those  sorties  and  put  them  in  sims  to  do  tactical  things,  could 
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have  a  5-10  year  shelf  life  of  this  guy,  give  a  one  month  transition 
and  they  are  ready  to  go.  Keep  them  “almost  ready  ”  and  save  a  ton 
of  sorties.  Unless  you  get  5-8  sorties  per  month,  they  can  be 
dangerous.  So  just  let  them  do  sims  and  keep  up  on  tactical  since  you 
need  to  do  the  retraining  for  piloting  anyway. " 

While  decay  might  be  an  issue  for  certain  skills,  the  specifics  vary  with  a  number  of  factors. 
These  include  the  following. 

1 )  The  experience  and  skill  of  the  pilot  before  rotation.  There  was  a  consensus  that  the  better  the 
pilot  was  during  active  flying,  the  easier  was  the  reentry  after  the  hiatus  staff  position.  However, 
this  effect  was  moderated  by  a  general  decrease  in  the  number  of  flying  hours  required  before  a 
pilot  could  rotate  out,  from  about  1000  hours  to  750.  There  was  a  consensus  that  rotating  back 
was  becoming  more  of  a  challenge  as  pilots  had  less  pre-rotation  experience.  The  importance  of 
skill  was  noted  in  comments  such  as  "It's  a  complete  bell  curve, "  "Some  guys  jump  right  back  in, 
some  guys  never  get  it  back"  and  "An  ace  before,  an  ace  afterwards. "  "A  middle  of  the  road  pilot 
is  still  a  middle  of  the  road  pilot. "  It  is  consistent  with  the  expertise  and  retention  literature  (e.g., 
in  studies  of  overlearning)  in  general,  that  the  greater  the  level  of  skill  development  before  a 
hiatus  the  higher  the  level  of  skill  at  the  end  of  the  hiatus  (greater  retention)  and  the  faster  the 
skill  will  recover. 

2)  The  nature  of  the  staff  position  in  relation  to  the  prior  and  subsequent  ones.  It  is  clear  that  the 
more  the  staff  position  resembles  or  is  related  to  the  prior  positions,  the  less  will  be  the  skill 
decline  (or  perhaps  interference).  However,  it  appears  that  the  needs  of  staffing  are  acute  enough 
that  such  targeted  assignment  is  not  often  feasible.  To  the  extent  possible,  it  would  be  desirable 
to  coordinate  staff  positions  with  prior  and  subsequent  duties,  so  that,  rather  than  interfering  skill 
retention  experiences  could  be  complementary — providing  a  different,  useful  perspective  on  the 
job. 

3)  The  opportunities  for  practice  and  refresher  during  the  staff  position.  There  was  a  consensus 
that  practice  or  refresher  activities  during  the  staffing  rotation  would  be  beneficial  for 
maintaining  skills,  but  that  duties  and  responsibilities  typically  make  it  difficult  to  impossible  to 
add  refresher  training  in  simulations  or  in  flight.  No  specification  was  offered  in  terms  of 
simulator  hours  but  it  was  acknowledged  that  skill  maintenance  activities  are  hard  to  squeeze 
into  a  hectic  pace  of  duties.  It  was  noted  that  different  kinds  of  activities  might  be  preferable  for 
different  kinds  of  skills,  e.g.,  computer  driven  exercises  for  keeping  current  on  checklists, 
procedures,  and  simulations  of  various  kinds  and  contacts  with  active  flyers  (e.g.,  through  blogs) 
for  keeping  up  with  tactics  and  equipment  advances. 

4)  The  nature  of  the  transition  (TX)  course  back  to  active  flying  duty.  There  are  structured 
reentry  experiences  for  those  rotating  out  of  a  staffing  position  back  to  more  front-line  services, 
including  flying  (Air  Force  Instruction  11-202  and  its  modifications;  see  U.S.  Air  Force,  2005). 
These  experiences  are,  in  some  cases,  tailored  to  the  prior  experience  of  the  person,  the  length  of 
time  away  from  flying,  and  the  new  job  to  be  assumed.  It  is  important  to  note  that  is  often  the 
case  that  the  new  position  does  not  involve  active  flying  of  the  same  sort  as  before  rotation.  The 
assignment  might,  for  example,  be  more  supervisory.  After  any  refresher  or  transition  training, 
the  nature  of  the  post-rotation  assignment  will  itself  continue  to  affect  the  level  of  degradation  of 
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skill  and  recovery.  A  key  factor  then  is  the  degree  to  which  pre-rotation  activities  and  those 
during  hiatus  are  consistent  with  the  demands  of  the  new  job  post  hiatus. 

An  argument  might  be  made  that  the  Air  Force  practices  work  against  the  acquisition  and 
retention  of  expertise.  As  one  interviewee  said, 

"When  the  Air  Force  finds  good  people,  they  have  them  do 
something  else,  since  they  are  the  future.  The  people  left  behind 
are  average,  but  they  get  more  experience.  By  the  time  they  make 
Major,  they  are  half  the  people.  The  Canadians,  British,  Dutch, 
they  do  not  have  the  up-or-out  system  we  do.  This  allows 
mentoring.  Some  captain  teaching  others,  a  job  he  has  done  for 
decades. " 


2.5.4  Differential  Decay 

Not  all  forms  of  skill  (reasoning  skill,  perceptual-motor  skill,  etc.)  decay  at  the  same  rate  or  in 
the  same  ways. 

In  an  Army  study,  Johnson  and  Cosby  (1985)  developed  a  prediction  instrument  based  on  input 
from  experienced  trainers,  who  decide  when  a  soldier  needs  training  and  then  plan  the  training. 
The  predictive  instrument  requires  the  soldier  trainer  to  rate  the  task  in  terms  of  complexity 
(steps,  time  limit,  mental  requirements).  From  the  ratings  an  overall  score  is  derived  for  each 
task,  and  is  used  to  predict  decay  rate  for  that  task.  As  one  would  surmise,  the  prediction  of 
decay  rates  based  on  judgments  of  experts  would  be  useful  in  predicting  decay  for  tasks  that  have 
no  accompanying  memory  aids,  that  are  complex  procedurally,  that  require  significant  amounts 
of  knowledge,  are  hard  to  remember,  and  also  have  significant  motor  skill  requirements — tasks 
that  are  likely  to  show  faster  decay. 

In  contrast,  a  classic  finding  in  the  motor  skills  literature  is  that  performance  after  learning 
simple  motor  skills  (e.g.,  accuracy  at  the  pursuit  rotor)  drops  by  only  about  25  percent  after  a 
year  (Bell,  1950).  Fleishman  and  Parker  (1962)  studied  performance  at  a  target-tracking  task 
(similar  to  the  pursuit  rotor  but  also  analogous  to  radar  tracking)  up  to  a  high  level  of  proficiency 
(integrated  error)  and  then  tested  performance  after  two  years  of  nonuse,  finding  very  little  loss 
of  skill  (i.e.,  return  to  highest  proficiency  after  only  about  five  trials).  The  primary  determinant 
of  performance  following  non-use  is  the  level  of  proficiency  achieved  up  to  the  point  that  the 
retention  interval  begins 

In  the  classic  studies  of  forgetting  by  Harry  Bahrick  and  his  colleagues,  they  showed  the  classic 
decay  curve  for  second  language  and  mathematics  learned  in  high  school,  but  that  decay  finding 
did  not  obtain  for  memory  of  names  and  faces,  over  intervals  that  ranged  up  to  57  years 
(Bahrick,  Bahrick,  &  Wittlinger,  1975). 

A  series  of  reports  by  Alice  F.  Healy  and  her  colleagues  to  the  Army  Research  Institute  for 
Social  and  Behavioral  Sciences  detailed  the  many  task  variables  that  influence  the  decay  of  skill 
learning  during  periods  of  nonuse.  Some  studies  examined  skill  retention  for  skills  having 
military  analogs,  others  used  tasks  representative  of  the  educational  (college)  context:  target 
detection  tasks;  circuit  design,  algebra  problem  solving,  data  entry  tasks,  memory  for  class 
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schedules,  vocabulary  learning.  Later  studies  in  the  ARI  project  looked  at  tank  gunner  skills, 
more  code  learning,  and  instrument  panel  scanning. 

The  findings  were  varied  and  mixed  both  with  regard  to  the  effects  of  the  key  variables  and 
individual  differences.  While  performance  following  hiatus  showed  a  decline  and  was  linked 
most  strongly  to  the  level  of  performance  achieved  just  prior  to  the  hiatus,  in  some  studies  and 
for  some  participants,  performance  following  a  hiatus  showed  improvement,  not  decay.  Skill  at 
multiplying  two  large  numbers  reached  a  point  of  near-automaticity  after  extensive  practice  and 
showed  almost  no  decay  after  a  seven-month  hiatus.  The  studies  of  vocabulary  learning  utilized 
the  mnemonic  technique  called  the  "keyword"  method,  which  relies  on  mental  imagery  and  is 
know  to  be  robust  to  memory  decay  (see  Hoffman  &  Senter,  1979).  A  number  of  studies 
conducted  in  this  ARI  project  found  surprising  levels  of  knowledge  and  skill  retention  (the  target 
detection,  multiplication  and  data  retention  tasks),  supporting  Bahrick's  notion  of  the 
"permastore." 

In  the  Henik,  et  al  (2001)  study  of  retention  of  gunnery  knowledge  and  skill,  a  "refresher  day" 
boosted  performance  on  the  declarative  memory  test  back  to  near  the  original  level,  but  had  no 
impact  on  performance  at  the  scenarios. 

We  know  that  perceptual-motor  skills  can  be  highly  specific  to  task  and  context.  Healy,  et  al., 
(2003)  cite  a  study  showing  that  performance  at  using  a  computer  mouse  is  significantly 
disrupted  by  changing  the  relation  between  mouse  movements  and  cursor  movements  (an 
example  of  negative  transfer  or  interference).  One  is  tempted  to  generalize  that  factual 
(“declarative”)  information  shows  poor  retention  but  robust  transfer  whereas  procedural 
knowledge  (skill)  shows  good  retention  but  limited  transfer.  Although  knowledge  can  be 
forgotten  very  rapidly,  learned  skills  are  usually  well  retained  across  periods  of  disuse.  Healy 
gives  the  example  of  learning  how  to  use  a  hula-hoop  as  a  child  and  able  to  use  a  hoop  again  well 
as  an  adult,  even  if  there  was  no  intervening  practice  with  the  hoop  since  childhood. 

The  differential  retention  of  procedural  skills  (perceptual-motor)  versus  declarative  knowledge  is 
noted  in  all  of  the  reviews,  but  the  inconsistencies  in  the  findings  make  generalization  tentative. 
One  possibility  is  that  simple  motor  skills  are  more  memorable  precisely  because  of  their 
simplicity,  and  that  even  the  more  complex  procedural  skills  that  have  been  studied  are  ones 
involving  task  procedures  can  be  put  to  rote  memory  (Healy,  et  al.,  1992). 

Within  the  general  "skills"  class,  it  has  also  been  noted  that  there  is  differential  retention  of  skills 
continuous  ("open-loop")  versus  relative  differential  decay  of  skills  that  are  discrete  ("closed- 
loop").  For  instance,  after  training  at  an  aircraft  simulator  people  will  tend  to  forget  checklist 
procedures  but  retain  skill  at  the  dynamic  continuous  task  of  actually  flying  the  plane 
(Mengelkoch,  Adams,  &  Gainer,  1971).  Some  studies  find  that  performance  at  closed-loop  tasks 
seems  to  decay  faster  but  a  meta-analysis  failed  to  affirm  this  generalization  (Arthur,  1998). 

As  the  above  discussion  suggests,  generalizations  about  decay/retention  are  difficult  to 
formulate.  One  cannot  conclude  that  perceptual  motor  skill  will  invariable  be  retained  better  than 
declarative  knowledge,  for  example.  Based  on  their  studies,  Healy  et  al.  generalize  that 
occasional  or  periodic  practice  or  refresher  work  during  hiatus  can  help  maintain  performance. 
There  is  some  evidence,  not  surprising,  that  the  refresher  practice  can  be  helpful  if  presented  just 
prior  to  the  critical  test  of  retention.  In  other  words,  refresher  practice  does  might  not  have  to  be 
distributed  across  a  hiatus  (Healy,  et  al.,  1995).  They  generalize  that  performance  evaluation  in 
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the  context  of  evaluating  decay  needs  to  be  conducted  in  a  way  that  reinstates  the  original 
learning  procedures  and  the  context  used  during  the  training. 


2.5.5  Generalizations  About  Decay/Retention 

Main  Generalizations 

Two  main  generalizations  can  be  asserted. 

(1) .  The  best  predictor  of  skill  retention  following  a  hiatus  is  the  level  of  performance  achieved 
(including  over-learning)  prior  to  the  hiatus.  This  finding  applies  generally,  including  to  military 
tasks  (see  Hagman  &  Rose,  1983). 

(2) .  The  variable  having  the  largest  impact  on  performance  after  a  retention  interval  is  the 
similarity  of  the  conditions  of  testing  to  those  of  the  training. 

The  results  of  this  study  suggest  that  the  similarity  of  the  training 
(acquisition)  and  work  (retention)  environments  plays  a  major  role 
in  the  retention  of  skills  and  knowledge  over  periods  of  nonuse... 
to  enhance  retention,  trainers  should  try  to  ensure  the  functional 
similarity  of  both  the  training  device  (acquisition)  and  actual  job 
equipment  (retention)  and  the  environment  in  which  both  are 
performed  (Arthur,  et  ah,  1998,  p.86). 

This  result  is  interesting  because  conditions-of-retrieval  is  directly 
related  to  the  issue  of  transfer  of  training.  Transfer  of  training  is 
the  generalization  of  trained  performance,  in  a  given  task,  from  the 
training  environment  to  the  work  environment  and  is  one  of  the 
key  criteria  for  evaluating  the  effectiveness  of  any  formal  training 
program  (Arthur,  et  ah,  1998,  p.86;  see  also  Kirkpatrick,  1987; 

Schendel,  Shields,  &  Katz,  1978). 

The  very  concepts  of  encoding  specificity  and  context  specificity  tend  to  play  against  notions  of 
transfer. 

Inconsistencies  and  Complications 

The  study  of  retention  is  complicated  by  a  potential  disconnect  between  the  skills  that  are  trained 
and  the  skills  that  are  actually  exercised  in  operational  performance.  For  instance,  in  their  study 
of  soldier  retention  of  equipment  maintenance  tasks,  Kneer,  et  al.  (1984)  noted  a  disconnect 
between  relation-the-job  ways  of  conducting  tasks  and  the  way  tasks  are  "taught  in  school,  " 
which  might  entail  negative  transfer.  Rose,  et  al  (1985)  presented  a  method  for  predicting  how 
fast  various  skills  will  decay  and  hence  need  retraining  (see  also  Johnson  &  Cosby,  1985).  The 
method  queries  learners,  for  example:  "Does  the  task  involve  a  job  or  memory  aid?"  The 
questions  refer  to  how  it  is  trained  and  tested,  rather  than  how  it  might  actually  be  done  (e.g., 
with  short-cuts,  embellishments,  work-arounds).  These  kinds  of  instruments  seem  to  test  for 
retention  of  training  material  and  methods,  rather  than  on-job  capability. 

Although  the  decay  function  usually  obtains  in  the  research,  for  both  movement  skills  and 
cognitive  tasks  (Arthur,  et  al.,  1998),  some  studies  show  good  levels  of  retention  of  declarative 
knowledge  in  the  months  and  years  following  learning.  A  study  by  Semb  and  Ellis,  (1992)  on 
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college  student  memory  for  material  learned  in  a  developmental  psychology  course  (85%-75% 
retention  after  four  months  and  80%-70%  retention  after  11  months).  Adding  to  this,  Semb,  et 
ah,  (1993;  see  also  Gaynor  and  Wolking,  1974)  found  that  student  tutors  retained  more  after  4 
months  than  the  students  they  tutored.  This  suggests  that  tutoring  has  its  own  learning  benefits. 

Even  the  phenomenon  of  over-learning  is  not  immune  from  qualification.  Rohrer,  et  al.  (2005) 
demonstrated  the  effect  of  over-learning  (additional  learning  trials)  for  students  who  learned 
geology  facts  and  word  definitions,  and  were  tested  one  and  nine  weeks  later.  While  the  over¬ 
learners  showed  better  recall  at  week  one,  by  week  nine  the  advantage  had  been  lost.  As  Arthur, 
et  al.  (1998)  pointed  out,  "[AJccuracy  asymptotes  rapidly  in  many  tasks,  leading  to  a  potentially 
false  conclusion  that  the  material  has  been  mastered...  [and]  because  learning  continues  past 
error-free  trials,  overlearning  may  really  just  be  a  higher  level  of  skill  acquisition"  (p.91;  see  also 
Regian  &  Schneider,  1990). 

Task  complexity  does  not  always  have  a  pronounced  effect.  In  an  analysis  of  skill  decay  in 
realistic  work  settings,  Lance  et  al.  (1998)  found  a  small  effect  of  task  complexity  but  in  a  review 
of  readiness  for  duty  of  reserve  recruits.  Wisher  (1999)  reported  that  task  complexity  (measured 
by  number  of  task  steps)  did  impact  the  degree  of  retention  over  longer  periods  of  time. 

Differences  in  inherent  organization  and  content  structure,  prior 
student  knowledge  and  experience,  task  requirements/demands,  and 
other  related  variables  may  contribute  to  these  discrepant  findings 
(Semb,  et  al.,  2003,  p.  315;  see  also  Arthur,  et  al.,  1998). 

Complexities  of  experimental  design  and  measurement  issues  (see  Arthur,  et  al.,  1998;  Semb,  et 
al,  1993;  Bacon  &  Stewart,  2006)  make  it  difficult  to  form  generalizations  that  might  be  carried 
over  to  the  consideration  of  accelerated  learning. 

Arthur,  et  al.  (1998)  point  out  that  retention  of  skill  involves  more  than  just  task  performance, 
since  other  forms  of  evaluation  are  possible  (e.g.,  worker  contribution  to  organizational  goals), 
and  furthermore  that  other  factors  can  influence  performance  in  the  operational  context.  Affect 
and  motivation  are  not  tapped  in  studies  that  look  exclusively  on  the  retention  and  decay  of 
knowledge.  This  is  an  important  consideration  since  high  levels  of  motivation,  that  is,  intrinsic 
motivation  to  work  on  hard  problems,  is  a  defining  feature  of  what  it  means  for  an  individual  to 
achieve  the  highest  levels  of  proficiency. 

A  seemingly  overwhelming  number  of  factors  are  in  play  in  the  training-performance  context. 
Classes  of  factors  are  task  factors  (e.g.,  difficulty,  representativeness),  practice  factors  (e.g., 
massed  versus  distributed  practice,  degree  of  overlearning),  acquisition  tasks  (motor  skill  versus 
cognitive,  procedures  versus  closed  loop  tasks),  test  tasks  (recall,  recognition),  trainee 
characteristics  (motivation,  intelligence),  feedback  type  and  timing,  and  retention  interval. 

In  research  on  retention/decay,  participants  are  often  not  tested  beforehand  to  assess  prior 
knowledge.  Previous  experience  or  knowledge  can  facilitate  long-term  skill  retention  even 
though  individuals  with  and  without  prior  experience  might  exhibit  the  same  level  of  skill 
immediately  after  training  (Bahrick,  1979).  It  is  possible  therefore  that  criterion  performance  at 
the  end  of  training  is  not  appropriate  for  use  as  a  predictor  of  long-term  retention. 

Since  the  shape  of  the  decay  function  is  so  dependent  on  multiple  interacting  variables  (Naylor  & 
Briggs,  1961)  training  for  any  particular  job  or  task  must  be  based  on  a  task  or  job-specific 
knowledge  of  the  course  of  forgetting  in  order  to  determine  the  best  plan  for  maintenance 
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training  or  re-training  (Arthur,  et  al.,  1998). 

The  experimental  psychology  of  learning  (retention/decay)  focuses  on  the  learning  and  retention 
of  tasks  (e.g.,  perceptual-motor  skills)  or  sets  of  content  (e.g.,  Spanish  learned  in  high  school). 
Arthur,  et  al.  note  that  the  study  of  skill  dec  ay /retention  for  tasks  that  involve  perceptual  learning 
and  perceptual  skill  (e.g.,  pattern  perception)  has  not  been  studied  much.  This  gap  in  the  research 
base  for  training  psychology  is  significant  since,  as  work  in  the  area  of  Expertise  Studies  has 
shown,  many  jobs  where  proficiency  is  required  and  expertise  is  desired  are  jobs  that  involve 
significant  perceptual  learning  and  perceptual  skills  (e.g.,  radiology)  (Klein  &  Hoffman,  1992). 

In  significant  domains  of  professional  practice,  including  in  the  military,  jobs  are  defined  by 
multiple  "tasks"  and  multiple  kinds  of  "tasks."  And  indeed  we  put  the  word  "tasks"  in  scare 
quotes  because  professional  jobs  do  not  consist  solely  of  the  performance  of  tasks  at  all,  in  the 
sense  of  a  stepwise  procedure  for  linking  means  to  ends.  In  actuality,  cognitive  work  involves  the 
context-sensitive,  knowledge-driven  choice  among  activities.  Professional  domains  involve  some 
tasks,  and  often  it  can  appear  as  if  the  practitioner  is  conducting  a  task,  but  this  will  be  when  the 
situation  is  routine  and  the  goal  is  well-defined. 


2.6  Team  Training 


In  most  controlled  studies,  experimenters  have  distinct  groups  of  participants  who  are  often 
called  novices  and  experts.  However,  in  a  real  work  environment,  individuals  at  various  levels  of 
experience  and  proficiency  are  always  working  together.  Workers  at  all  proficiency  levels  can  be 
accountable  for  one  another’s  actions,  they  can  give  instructions  as  tasks  are  carried  out,  they  can 
devise  coordinated  solutions  that  mix  knowledge  and  skill  contributions,  and  they  are  often 
bound  in  social  and  work  role  hierarchies  (Lajoie,  2003). 

The  generalizability  of  findings  from  laboratory  studies  or  from  research  using  non-military 
teams  may  be  an  open  question.  However,  a  recent  meta-analysis  on  "team  training"  (Delise,  et 
al.,  2010)  found  evidence  that  there  were  no  differences  between  civilian  and  military  teams,  nor 
were  there  differences  across  research  settings  (i.e.,  lab  or  field).  Further,  there  were  no 
differences  between  ad  hoc  and  intact  teams.  Because  both  civilian  and  ad  hoc  teams  in  the 
studies  comprising  the  meta-analysis  were  likely  college  students,  this  suggests  that  findings 
from  this  literature  may  generalize  to  studies  concerning  how  to  accelerate  learning  in  teams. 
Importantly,  with  regard  to  measures  related  to  transfer,  although  there  were  some  differences  in 
cognitive  outcomes  based  upon  the  setting  of  the  evaluation  (lab  or  field  tests),  there  were  none 
for  task  or  teamwork  skills.  They  note  that  this  suggests  that: 

[Tjraining  is  similarly  effective  in  both  proximal  and  distal 
application  situations,  except  with  cognitive  outcomes.  Most 
critical  to  the  present  purposes,  analyses  indicated  that  training  had 
large  positive  effects  on  cognitive  outcomes  for  training  and 
transfer  outcomes,  but  the  effects  were  larger  for  transfer  outcomes 
than  for  training  outcomes  (Delise,  et  al.,  2010,  p.  70). 
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2.6. 1  Complex  Cognitive  Work  is  Teamwork 

This  Report  has  emphasized  proficiency  in  domains  of  "complex  cognitive  work."  Such  domains 
are  also  called  "sociotechnical  systems"  in  recognition  of  the  fact  that  the  work  is  conducted  by 
collaborations  of  people  and  technologies  (see  Hoffman  &  Militello,  2009). 

A  study  of  expert  software  teams  by  Sonnentag  (2000)  showed  that  experienced  problem  solvers 
place  a  high  value  on  cooperation  and  engage  in  more  work-related  communication.  The 
problem  solving  process  in  many  complex  domains  relates  clearly  and  directly  to  team  cognition 
research  in  areas  such  as  software  development,  system  administration,  and  design  teams  -  these 
range  from  somewhat  homogeneous  teams,  such  as  in  software  development,  to  often 
heterogeneous  teams  such  as  in  systems  administration,  but  these  teams  are  all  created  to 
develop,  manage,  and  maintain  some  complex  technological  product  or  system. 

Teamwork  is  necessary  for  many  aspects  of  the  problem  solving  process,  and  team  members 
need  to  be  trained  accordingly.  First  there  is  a  need  to  recognize  that  a  problem  has  occurred 
followed  by  the  need  to  define  and  represent  that  problem,  culminating  in  discussions  about 
generation  and  testing  of  particular  solutions.  Problem  recognition  can  be  an  individual  or  a  team 
task.  Critical  to  subsequent  problem  solving  processes  is  information  sharing,  in  that  the 
heterogeneous  nature  of  the  team’s  knowledge  and/or  the  particular  roles  of  the  team  members 
require  efficient  sharing  of  data  based  on  each  team  member's  understanding  of  the  needs  of  the 
other  team  members.  The  distributed  nature  of  the  work  sometimes  means  that  information  from 
differing  sources  needs  to  be  integrated,  and  this  is  often  a  team  task  rather  than  an  individual 
task. 

Many  collaborative  tasks  consist  of  a  dynamic  social-cognitive  process  requiring  diagnostic 
interrogation  of  some  system  and  diagnostic  questioning  from  an  often  times  ad-hoc  team.  Haber 
(2005)  refers  to  this  as  “group  sense  making.” 

A  problem  existed  due  to  interactions  between  the  components  of  a 
very  complicated  system,  and  the  experts  on  the  different 
components  needed  to  work  together  to  understand  the  cause  and 
find  a  solution.  The  overall  strategy  was  a  cycle  of  shared 
observations  of  the  system  in  question,  developing  hypotheses  as 
individuals,  small  groups,  or  the  group  as  a  whole,  and 
implementing  changes  to  attempt  a  fix  (p.  3). 

Maglio,  et  al.  (2003)  similarly  discuss  computer  systems  administration  from  a  perspective  of 
complex  problem  solving,  also  noting  the  social-cognitive  nature  of  the  collaboration.  They 
describe  a  requirement  for  developing  common  ground  and  the  coordination  of  attention  across  a 
number  of  team  members,  ranging  from  engineers  engaged  in  trouble-shooting,  technical  support 
personnel,  and  software  application  developers. 

A  crucial  part  of  this  process  is  the  emphasis  on  cooperation  strategies  that  are  utilized  in  system 
administration.  The  work  places  high  cognitive  demands  on  practitioners. 

As  system  administrators  troubleshoot  systems,  making  sense  of 
millions  of  log  entries  by  controlling  thousands  of  configuration 
settings,  and  performing  tasks  that  take  hundreds  of  steps.  The  work 
also  places  high  social  demands  on  practitioners — as  systems 
administrators  need  organizational  and  interpersonal  skills  to 
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coordinate  tasks  and  collaborate  effectively  with  others  (Barret,  et 
al  2004). 

Related  to  this,  Sonnentag  and  Lange  (2002)  found  that  among  engineering  and  software 
development  teams,  a  general  knowledge  of  cooperation  strategies,  that  is,  what  to  do  in 
situations  requiring  cooperative  behavior,  is  related  to  better  performance.  Cooperation  is  more 
valued  by  the  experts  than  by  the  moderate-level  performers.  Experts  engage  in  higher  amounts 
of  work  related  communication,  helped  their  coworkers,  and  sought  out  feedback  from 
coworkers  (Sonnentag,  2000).  Another  finding  is  that  teamwork  improves  following  training  in 
which  teams  are  given  prompt  meaningful  feedback  about  how  they  coordinate  (Salas,  Nichols, 
and  Driskell,  2007). 

The  team  leader  can  play  a  critical  role  in  complex  problem  solving  teams-enabling  (or 
obstructing)  information  sharing  and  coordination.  The  team  structure  and  conflicting  team 
member  goals  and  roles  can  sometimes  impede  problem  solving  (Haber,  2005). 

We  refer  to  this  collaboration  pattern  as  “Seven  People,  One 
Command  Line,”  as  various  people  participated  in  troubleshooting, 
but  only  George  had  access  to  the  system.  His  manager  wanted  to 
know  when  the  problem  would  be  fixed  and  whether  others  should 
be  redirected  to  help  him  complete  the  task  on  time.  The  support 
person  wanted  to  resolve  the  problem  ticket  and  end  the  call  as 
quickly  as  possible.  His  colleague  wanted  to  help  within  the 
limitations  imposed  by  his  own  responsibilities.  The  system 
architect  wanted  to  know  if  there  was  any  problem  in  the  overall 
design  without  being  mired  in  the  details.  Other  specialists  waited 
for  instructions  to  manipulate  the  subsystems  they  were 
responsible  for.  The  problem  was  eventually  found  to  be  a  network 
port  misconfiguration.  George  misunderstood  the  meaning  of  a 
certain  configuration  parameter  for  the  new  web  server.  George’s 
misunderstanding  affected  the  remote  collaborators  significantly 
throughout  the  troubleshooting  session  (p.  4). 

Finally,  the  team  leader  can  play  an  important  role  in  complex  problem  solving  teams  -  enabling 
[or  obstructing]  information  sharing  and  coordination.  For  example,  from  the  standpoint  of 
teamwork  and  problem  solving,  Haber  (2005)  noted  how  the  team  structure  and  conflicting  team 
member  goals  and  roles  can  sometimes  impede  problem  solving. 

2.6.2  Coordination 

Nancy  Cooke  and  her  colleagues  have  conducted  a  great  deal  of  research  on  team  coordination  in 
command- and-control  environments,  especially  focused  on  the  development  of  team 
coordination  skills  (Cooke,  et  al.,  2007).  The  authors  characterize  team  command-and-control 
tasks  as  challenging  because  they  are:  1)  defined  by  unanticipated  situations,  2)  often  involve  ad- 
hoc  team  structure,  3)  team  members  may  not  be  familiar  with  each  other,  4)  little  team  training 
occurs  over  some  time  periods  (i.e.,  stretches  of  time  without  further  training).  The  researchers 
have  primarily  addressed  the  third  and  fourth  issues  within  the  context  of  Unmanned  Air  Vehicle 
(UAV)  command-and-control.  This  effort  was  conducted  in  three  parts:  an  experiment 
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examining  skill  acquisition  and  retention,  the  development  of  a  model  of  optimal  coordination, 
and  a  follow-up  experiment  comparing  types  of  training. 

The  model  of  coordination  developed  by  Cooke,  et  al.  is  a  synthesis  of  two  existing  theoretical 
models:  the  procedural/stage  theory  of  coordination  and  the  dynamical  systems  theory  of 
coordination.  The  procedural  theory  states  that  teams  execute  a  shared  plan  from  a  point  of 
common  understanding  in  order  to  achieve  a  common  goal.  The  authors  offer  Klein's  (2001) 
example  of  the  stages  involved  in  selecting  an  air  strike  package:  Preparation,  Planning, 
Direction,  Execution,  and  Assessment.  While  these  stages  are  explicitly  defined  by  the  theory, 
the  authors  note  that  they  may  occur  implicitly  during  team  interactions  rather  than  as  overt 
actions.  Under  the  dynamical  systems  theory  of  coordination,  teams  self-organize  in  order  to 
reach  their  goal.  This  theory  states  that  there  is  no  a  priori  script  or  plan  that  team  members 
share,  and  their  coordination  results  from  their  interactions  with  each  other  and  the  task 
environment. 

The  first  experiment  was  conducted  using  the  UAV-STE  (Synthetic  Task  Environment) 
developed  by  the  authors.  The  authors  examined  the  acquisition  and  retention  functions  of  teams 
based  on  manipulations  of  Retention  Interval  Eength  (i.e.,  the  time,  short  or  long,  until 
participants  engaged  in  a  follow-up  session)  or  and  Team  Composition  (intact  or  mixed).  Teams 
were  trained  an  tested  together,  and  then  were  asked  to  participate  in  a  follow-up  session.  If  a 
team  was  “mixed,"  the  follow-up  session  was  conducted  without  keeping  the  initial  team  intact. 
The  researchers  found  that  mixed  teams  (i.e.,  teams  whose  members  were  not  familiar  with  each 
other)  displayed  a  significant  performance  decrement  in  the  follow-up  session,  while  those  that 
were  intact  did  not  have  such  a  decrement.  The  length  of  time  until  participant  engaged  in  a 
follow-up  session  (i.e.,  short  =  3  to  6  weeks,  long  =  10  to  13  weeks)  had  some  effect  on 
performance,  with  short-intact  teams  having  a  lower  performance  deficit  than  other  teams. 
However,  the  performance  deficit  due  to  retention  disappeared  after  the  second  mission  in  the 
follow-up  session.  While  performance  was  lower  in  mixed  teams,  the  individuals  who  had  mixed 
team  sessions  showed  greater  improvements  in  knowledge  accuracy  from  session  1  to  session  2. 
However,  short-intact  teams  showed  consistent  improvement  in  teamwork  knowledge  measures 
from  session  1  to  session  2.  The  authors  suggest  that  these  results  indicate  that  mixing  team 
composition  results  in  short-term  performance  decrements  but  with  long-term  teamwork  and 
taskwork  knowledge  gains. 

In  the  second  experiment,  team  mental  models  were  examined  using  the  UAV-STE.  As  in  the 
first  experiment,  participants  engaged  in  an  initial  session,  with  a  follow-up  session  some  time 
after  (8  to  10  weeks  after  the  first  session).  Team  training  was  manipulated  by  the  introduction  of 
perturbations,  which  took  the  form  of  static  introduced  over  communication  lines.  The  three 
team  conditions  were  procedural  training,  cross-training,  and  perturbed  training.  The  authors 
found  no  significant  differences  in  teamwork  between  the  two  sessions  or  the  team  conditions. 
Increased  workload  during  the  experiment  resulted  in  significant  decreases  in  coordination  for  all 
team  types.  However,  procedural  teams  were  slower  to  overcome  roadblocks  (i.e.,  problems 
during  the  task)  than  cross-trained  and  perturbed  teams.  The  authors  suggest  that  perturbed  teams 
were  the  highest  performers,  though  the  effects  on  coordination  were  not  sufficient  to  draw 
conclusions  regarding  coordination. 
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2.6.3  Requirements  for  Team  Training 

As  has  often  been  said,  a  team  of  experts  is  not  necessarily  an  expert  team.  High  proficiency  on 
the  part  of  individuals  with  respect  to  the  goals  and  requirements  of  the  tasks  of  individuals  is 
certainly  desired  for  all  team  members.  But  equally  important  is  skill  at  teamwork  and  team 
tasks,  such  as  coordination  and  the  development  of  shared  mental  models. 

First,  with  regard  to  how  to  prepare  individuals  for  team  training,  what  is  required  is  clearly 
explicating  the  precursors  to  team  training  in  the  training  of  the  individuals.  Doing  so  requires 
attending  to  the  systematic  techniques  of  training  needs  analysis  which  examines  three  inter¬ 
related  levels  to  account  for  the  multitude  of  issues  affecting  the  process  and  performance  issues 
experienced  by  teams  (see  Salas,  Sims,  Klein,  &  Burke,  2003).  At  the  organizational  level,  this 
analysis  identifies  whether  or  not  the  institution  supports  training  implementation  and  helps  to 
identify  where  and  when  training  is  needed.  At  the  level  of  the  task,  cognitive  task  analysis  and 
team  cognitive  task  analysis  (e.g.,  Crandall,  Klein,  &  Hoffman,  2005;  Klein,  2000),  must  be  used 
to  identify  the  training  content.  Finally,  a  team  analysis  is  necessary  to  detail  who  within  the 
team  needs  training  and  the  degree  to  which  team  process  or  individual  training  is  needed  (e.g., 
skills-based  training  at  an  individual  level  or  communication  training  at  the  team  level).  Critical 
to  this  level  of  analysis  is  articulating  the  interdependencies  within  the  team  so  as  to  lay  out  the 
coordination  requirements  to  be  trained. 

Second,  with  regard  to  what  to  train,  team  training  must  consider  content  knowledge,  perceptual- 
motor  skills,  reasoning  skills  and  strategies,  coordination  and  collaboration  skills,  and  attitudes 
appropriate  for  teamwork.  New  notions  such  as  that  of  shared  mental  models  and  transactive 
memory  systems  are  relevant  to  accelerating  the  development  of  proficiency  in  teams. 

"Shared  mental  model"  is  the  term  used  to  describe  knowledge  structures  held  by  team  members 
which  are  related  to  understanding  the  task,  their  teammates,  and  their  teammates'  roles  (Cannon- 
Bowers,  Salas,  &  Converse,  1993).  The  recent  introduction  of  this  idea  in  the  field  of  human 
factors  represents  an  important  advance  for  our  ability  to  analyze  team  cognitive  work,  but  as 
with  all  concepts  of  complexity,  it  needs  qualification.  Within  the  work  of  teams  are  many 
individual  tasks  and  goals  that  are  the  responsibility  of  an  individual  and  involve  activities  and 
knowledge  that  one  or  more  of  the  other  team  members  does  not  need  to  know.  A  surgical  nurse 
may  need  to  track  the  patient's  blood  pressure  but  the  surgeon  may  need  to  know  only  that  the 
pressure  is  at  some  acceptable  level.  The  nurse  may  need  to  be  able  to  anticipate  that  the  surgeon 
will  soon  need  a  particular  clamp,  but  the  surgeon  does  not  need  to  know  where  the  nurse  has  to 
go  to  get  that  clamp.  To  say  that  there  is  a  "shared"  or  "team  mental  model"  does  not  mean  that 
all  team  members  know  all  the  same  things  Any  approach  aimed  at  providing  all  the  information 
and  knowledge  to  everyone  on  a  team  is  certainly  misguided. 

Wagner  (1986)  “defined  the  transactive  memory  system  as  a  combination  of  the  knowledge 
possessed  by  each  individual  and  a  collective  awareness  of  who  knows  what”  (Austin,  2003,  p. 
866).  Transactive  memory  is  used  to  help  team  members  have  a  sense  of  teammate  expertise. 

“Team  situation  awareness”  (SA)  is  another  useful  concept  for  understanding  teamwork.  But 
again,  there  is  the  qualification:  The  concept  of  team  SA  should  not  be  taken  to  mean  that  at  any 
one  moment  all  of  the  team  members  need  to  be  aware  of  all  of  the  same  factors  in  their 
immediate  task  environment.  Again,  any  approach  aimed  at  providing  all  the  information  and 
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knowledge  to  everyone  on  a  team  is  certainly  misguided.  Endsley  and  Robertson  (2000)  describe 
a  Team  Situational  Awareness  Training  Course,  based  on  five  major  SA  issues  in  aviation 
maintenance:  1)  incongruous  mental  models,  2)  lack  of  communication,  3)  lack  of  feedback,  4) 
lack  of  teamwork  and  shift  meetings,  and  5)  task  and  environment  issues.  Their  evaluation  of  the 
training  determined  the  perceived  value  and  usefulness  of  the  training,  the  training's  effects  on 
attitude  and  on  actual  workplace  behavior.  In  all  of  these  categories  of  evaluation,  the  training 
intervention  was  rated  (by  the  trainees)  as  useful  or  effective,  and  follow-up  measures  indicated 
that  workplace  activities  and  teamwork  changed  after  the  training  administration. 

With  regard  to  team  skills,  communication  (Cannon-Bowers,  Salas,  Tannenbaum,  &  Volpe, 
1995;  Salas,  Burke,  &  Cannon-Bowers,  2000)  is  key  to  process  and  performance.  Stated  most 
simply,  this  is  effectively  exchanging  information  “between  two  or  more  team  members  in  the 
prescribed  manner  and  with  proper  terminology  [along  with]  the  ability  to  clarify  or 
acknowledge  the  receipt  of  information”  (Cannon-Bowers  et  ah,  1995,  p.  344).  It  involves 
information  sharing  and  closed  loop  communication  as  well  as  volunteering/requesting 
information. 

Coordination  is  a  skill  also  critical  to  performance  outcomes.  Marks,  Mathieu,  and  Zaccaro 
(2001)  defined  this  as  “the  process  of  orchestrating  the  sequence  and  timing  of  interdependent 
actions”  (p.  367). 

Finally,  important  team  attitudes  for  training  include  psychological  safety,  that  is,  a  team 
member’s  perception  that  his/her  team  is  safe  for  interpersonal  risk  taking.  (Edmonson,  1999) 
and  collective  efficacy,  defined  as  the  team’s  belief  they  have  the  capability  to  complete  their 
tasks  (Salas,  Burke,  &  Cannon-Bowers,  2000). 

There  is  a  significant  motivational  component  to  team  training  just  as  there  is  to  individual 
training.  Instruction  that  that  is  useful  in  training  for  complex  cognitive  work  prompts  the  team 
members  to  approach  learning  as  effortful  and  mindful  ("active  learning")  but  at  the  same  time  a 
collaborative.  This  is  generally  associated  with  improved  skill  acquisition  and  increased 
adaptivity.  As  for  individuals,  teams-in-training  need  to  focus  on  mastering  tasks,  rather  than 
getting  individual  problems  right  ("mastery  training").  Individual  team  members,  and  teams  as  a 
whole  need  to  have  a  sense  of  achievement  to  balance  a  sense  of  failure,  which  might  be  more 
impactful  because  the  work  is  cognitively  difficult  and  hence  trainees  might  be  especially  prone 
to  failure  in  the  early  phases  of  instruction.  Another  finding  is  that  feedback  to  individual  team 
members  works  better  if  provided  to  all  the  members  of  the  team,  in  the  team  context.  "Research 
indicates  pervasive  effects  for  the  motivational  components  on  self-regulatory  processes, 
learning,  and  performance"  (Kozlowski  &  DeShon,  2005,  p.  6-7). 


2.6.4  Methods  for  Team  Training 

The  studies  making  up  the  Delise  et  al.  meta-analysis  included  a  variety  of  training  methods  such 
as  lecture-based  training,  group  discussion,  videos  and  task  demonstrations  and/or  practice. 
Other  methods  include  those  designed  to  incorporate  training  content  that  encompasses  a  broad 
swath  of  learning  requirements;  that  is,  incorporates  knowledge,  skills,  and  attitudes.  Both  event- 
and  scenario-based  training  (e.g.,  Oser,  Gualtieri,  Cannon-Bowers,  and  Salas,  1999;  Salas  & 
Cannon-Bowers,  2000)  are  relevant  methods  for  capturing  the  complexities  inherent  in  high- 
levels  of  performance.  Here  complex  scenarios  are  developed  to  elicit  targeted  behaviors,  thus 
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providing  contextual  grounding  for  learning  and  understanding.  From  the  standpoint  of  continual 
learning,  self-correction  and  guided  self-correction  training  are  effective  in  helping  teams 
continue  developing  their  expertise  (e.g.,  Smith-Jentsch,  Zeisig,  Acton,  &  McPherson,  1998). 
These  methods  teach  team  members  to  evaluate  their  own,  and  their  teammate’s,  behaviors  so  as 
to  judge  their  effectiveness.  Relevant  for  high-performing  teams  is  also  research  in  team 
coordination  training  which  combines  knowledge  with  behavior  to  support  team-level 
understanding  in  how  to  sequence  and  integrate  actions  (Entin  &  Serfaty  1999).  Also  important 
is  work  on  cross-training  which  is  designed  to  support  development  of  shared  understanding  and, 
more  critically,  shared  expectations,  by  training  the  roles  of  teammates  (for  a  review  see  Stagl, 
Salas,  &  Fiore,  2007).  Finally,  as  for  use  of  simulation-based  training,  recent  developments  may 
enable  a  more  efficacious  implementation  of  learning  and  training.  For  example,  simulation- 
based  handoff  scenarios  are  designed  to  specifically  focus  teams  on  communication  and  handoffs 
(Berkenstadt  et  ah,  2008).  Other  methods  incorporate  embedded  instructional  agents  to  monitor 
and  assess  trainees  (e.g.,  Neville,  Sorensen,  Glucroft,  &  Barba,  2002;  Ryder,  Santarelli,  Scolaro, 
Hicinbothom,  &  Zachary,  2000). 

Team  Training  Using  Simulations 

Fletcher  (1999)  approached  the  subject  of  team  training  via  networked  computer  simulation 
applications.  Following  a  basic  definition  of  simulations,  he  describes  how  simulations  have 
been  used  in  the  military,  noting  the  increasing  popularity  of  simulations  for  both  training  and 
assessment.  Because  of  this  popularity,  Fletcher  notes,  the  psychology  community  has  started  to 
approach  simulations  from  the  standpoint  of  psychometrics  and  measurement  in  general.  In  other 
words,  simulations  are  now  discussed  in  terms  of  reliability,  validity,  and  precision.  This  turn 
toward  using  simulation  for  measurement  is  meaningful  because  it  addresses  a  specific  need  of 
the  military  community:  combat  skill  assessment.  While  Fletcher  argues  that  comprehensive 
combat  skill  assessment  is  impossible  (due  to  its  requiring  actual  wartime  activity),  simulation 
allows  for  training  and  evaluation  of  military  forces  in  situations  that  are  not  feasible  to 
reproduce  in  reality  (i.e.,  virtual  wartime  conditions).  Fletcher  proposes  that  readiness  be  used  as 
a  surrogate  for  combat  effectiveness,  as  readiness  can  be  trained  and  evaluated  without  war, 
whereas  combat  effectiveness  requires  actual  combat  conditions.  As  an  example  of  the  efficacy 
of  readiness  training,  Fletcher  offers  the  applied  case  of  Navy  vs.  Air  Force  training  during  the 
lull  in  the  North  Vietnam  air  war.  Simulated  combat  allowed  Navy  pilots  to  outperform  Air 
Force  pilots  who  did  not  receive  combat  simulation  training.  However,  this  data  is  non- 
experimental,  and  only  suggest  (rather  than  prove)  efficacy. 

Following  this  discussion  of  simulation  and  combat  readiness,  Fletcher  describes  tactical  teams, 
which  are  groups  of  individuals  that  make  operational  decisions.  These  teams  vary  in  size  and 
scope,  and  are  effectively  decision  making  or  action  teams  in  the  wider  psychological  literature. 
Fike  other  teams,  these  tactical  teams  must  make  decisions  in  real  time  under  a  variety  of  outside 
constraints  and  pressures. 

After  a  brief  discussion  of  shared  mental  models  in  teams,  Fletcher  arrives  at  the  core  of  his 
discussion:  the  use  of  networked  military  simulation  to  train  and  assess  decision  making.  As  a 
key  point,  Fletcher  points  out  that  simulations  are  not  an  inferior  replacement  to  real-world 
training,  but  are  in  actuality  a  tool  that  allows  trainers  and  trainees  to  surpass  the  limitations  of 
reality  and  provide  information,  opportunities,  and  context  that  would  otherwise  be  unavailable. 
Simulations  are  broken  down  into  three  primary  categories:  live  simulations,  constructive 
simulations,  and  virtual  simulations.  Five  simulations  involve  experience  in  the  field,  using 
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actual  equipment.  Constructive  simulations  are  non-interactive  models  of  reality  (i.e.,  the 
“modeling”  component  of  modeling  and  simulation).  Virtual  simulations  are  computer-based 
simulations  that  are  interactive,  providing  a  digital  version  of  live  simulation.  No  one  simulation 
category  is  superior  to  another,  as  they  all  provide  different  information  and  afford  different 
types  of  training. 

Given  that  the  military  is  a  large  organization  consisting  of  teams  of  teams,  simulation  in  the 
military  must  provide  the  organization  with  opportunities  to  train  with  large  numbers  of 
individuals.  Networked  simulations  allow  collective  proficiency  to  be  trained,  and  Fletcher  notes 
that  they  are  especially  applicable  to  such  applications  because  they:  1)  focus  on  groups,  2)  allow 
for  distributed  training,  3)  work  in  real  time,  4)  allow  for  dynamic  environments  and  tasks,  5) 
provide  performance  data,  6)  are  affordable,  7)  can  be  high-fidelity,  and  8)  allow  for  multi-unit 
interactions.  However,  the  use  of  networked  simulations  for  problem  solving  and  decision 
making  training  requires  effective  performance  measures,  which  must  be  diagnostic, 
discriminative,  valid,  and  reliable.  Following  a  description  of  the  criteria  that  such  measures 
must  meet,  Fletcher  describes  the  specific  problem  domain  of  problem  solving  and  decision 
making,  offering  an  overview  of  the  processes  involved  therein. 

Schreiber,  et  al.  (2009)  present  the  history  of  use  of  flight  simulators  for  training  and  assessment, 
from  its  beginnings  in  about  1910  to  the  present,  including  current-day  activities  that  enable 
practice  on  "distributed  mission  operations"  (DMOs).  These  are  team-based  training  involving 
fairly  elaborate  scenarios  and  activities  of  a  complex  operation.  There  are  many  challenges  of 
assessment  of  performance  and  proficiency  levels  in  these  new,  elaborate,  multi-person,  high- 
technology  environments  that  are  situation  reactive  and  capable  of  many  manipulations. 
Assessment  as  currently  conducted  relies  heavily  on  subjective  evaluation,  which  Schreiber,  et 
al.  feel  have  limitations.  There  is  an  outstanding  need  for  good  team  performance  assessment 
measures  and  techniques  in  these  environments,  as  indicated  by  the  many  current  Department  of 
Defense  research  programs  aimed  at  issues  of  team  performance  metrics. 

Accelerated  Learning  for  Teams? 

The  acceleration  of  team  proficiency  will  require  its  own  instructional  strategy  (Salas  &  Cannon- 
Bowers,  1997,  p.  313)  although  just  as  for  individual  training,  it  will  presuppose  the 
identification  of  the  requisite  knowledge,  skills,  and  attitudes  associated  with  the  given  training 
need,  and  match  these  to  the  methods  appropriate  for  this  need  (Salas  &  Cannon-Bowers,  2000). 
Also  critical  for  any  effort  in  accelerating  team  proficiency  is  matching  the  method  of  training 
with  the  prior  experience  of  the  team  members — experience  at  the  domain  tasks,  at  the  team 
tasks  (which  differ  from  the  tasks  of  individuals)  and  experience  at  being  a  team  member. 


2.6.6  Some  Generalizations? 

•  Individuals  who  are  more  proficient  at  solving  problems  related  to  the  team's  primary 
goals  (team  task  work)  place  a  high  value  on  cooperation  and  engage  in  more  work- 
related  communication  (team  work).  Thus,  training  in  the  achievement  and  maintenance 
of  common  ground  will  enhance  team  performance. 

•  In  high  performing  teams,  the  team  members  develop  rich  mental  models  of  the  goals  of 
the  other  team  members.  Training  that  promotes  cross-role  understanding  facilitates  the 
achievement  of  high  proficiency  in  teams. 
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•  High  performing  teams  effectively  exchange  information  in  a  consensually  agreed-upon 
manner  and  with  consistent  and  meaningful  terminology,  and  are  careful  to  clarify  or 
acknowledge  the  receipt  of  information.  Training  that  promotes  such  communication 
facilitates  the  achievement  of  high  proficiency  in  teams. 

•  Practice  while  team  membership  changes  can  facilitate  the  acquisition  of  team  work 
skills. 

•  The  question  of  performance  decay  in  team  work  skills  due  to  hiatus  is  an  open  subject 
for  research. 

•  Training  for  accelerated  team  proficiency,  like  training  for  individuals,  benefits  from 
problem  or  scenario-based  practice  using  realistic  and  complex  scenarios  (e.g.,  serious 
games,  simulated  distributed  mission  operations). 

It  makes  sense  to  seek  to  form  and  maintain  teams  in  terms  of  consistency  of  team  membership, 
giving  the  team  a  chance  to  practice  team  work  and  become  an  "expert  team,"  just  as  the  musical 
quartet  must  practice  extensively  in  order  to  develop  a  capacity  for  collective  improvisation. 
However,  as  the  findings  of  Cooke  et  al.  show,  the  mixing  of  teams  can  have  its  own  benefits  in 
that  it  allows  individuals  to  gain  knowledge  and  work  on  their  team  work  skill,  that  is,  ability  to 
forge  a  high-proficiency  team.  In  addition,  in  military  teams  membership  is  often  necessarily  ad 
hoc,  there  is  a  mixture  of  proficiency  levels,  and  team  membership  typically  changes  frequently. 

In  the  case  of  teamwork  and  team  training,  generalizations  are  hard  to  come  by  primarily  because 
the  study  of  complex  cognitive  work  by  teams  is  a  relatively  new  area.  It  is  certainly  open  for 
investigation  and  may  benefit  by  attempting  to  devise  rapidized  training  or  other  forms  of 
acceleration.  In  the  next  section  we  discuss  the  literature  on  accelerated  learning,  what  little  there 
is,  and  we  see  there  that  many  of  the  demonstrations  of  acceleration  are  in  fact,  focused  on  teams 
and  team  work. 


2.7  Accelerated  Learning 

The  first  reference  we  have  found  to  the  notion  of  accelerated  learning  is  a  paper  by  Carlson 
(1990)  in  reference  to  expertise  in  the  mental  health  care  industry.  Carlson's  paper  was  a 
proposal,  based  on  the  finding  of  great  variation  in  the  effectiveness  of  mental  health  services 
(for  depression  alcoholism,  etc.)  across  counselors,  health  facilities,  geographical  regions,  etc. 
Carlson  invoked  the  Dreyfus  and  Dreyfus  (1986)  developmental  stages  (see  Tables  2.2  in  the 
present  Report)  arguing  that  health  care  workers  needed  to  be  trained  beyond  the  use  of  fixed 
procedures  and  rules,  up  to  the  stage  where  reasoning  is  context-sensitive.  Carlson  proposed  that 
this  situation  could  be  improved  if  health  care  analysis  were  based  not  on  patient  throughput,  but 
on  the  identification  of  genuine  expertise  with  regard  to  patient  care.  Carlson  proposed  an 
attempt  to  instill  that  expertise  through  feedback-based  training  utilizing  scenarios  created 
through  consultation  with  highly  experienced  clinicians. 

This  notion  of  accelerated  learning  highlights  an  important  theoretical  and  practical  tension.  On 
the  one  hand  is  the  notion  that  we  must  accelerate  learning;  that  is,  increase  the  speed  with  which 
proficient  performance  can  be  acquired.  On  the  other  hand,  there  is  a  significant  amount  of 
evidence  that  developing  expertise  requires  up  to  10  years  of  experience  and  practice  with  zeal 
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including  practice  at  "tough  tasks"  (Hoffman,  1998),  suggesting  that  it  is  not  possible  to 
accelerate  the  achievement  of  high  proficiency. 

In  this  section  of  this  Report  we  present  a  set  of  case  studies  that  report  training  that  can  be 
counted  as  successful  demonstrations  of  acceleration. 


2. 7. 1  Accelerated  Learning  in  Business 

A  general  theme  in  the  field  of  knowledge  management  is  that  learning  on  the  part  of  corporate 
employees  might  be  accelerated  through  social  networking.  One  reason  is  that  information 
sharing  can  help  prevent  decision  errors  (cf.  Townsend,  2009).  Social  networks  derived  by 
sociometric  analysis  (who  talks  to  whom,  who  shares  information  with  whom,  who  asks 
questions  of  whom)  can  show  where  the  corporate  knowledge  and  expertise  lies  within  the 
organization  (Becerra-Fernandez,  et  ah,  2004,  2008;  Stein,  1992,  1997).  Expert  judgments  of  the 
competencies  of  other  experts  conform  to  evidence  from  experts’  retrospections  about 
previously-encountered  critical  incidents.  Competency  judgments,  and  competencies  shown  in 
incident  records,  can  be  used  to  predict  practitioner  performance  (McClelland,  1998).  There  is 
ample  evidence  that  social  networking  tools  and  capabilities  (Blogs,  Facebook,  Twitter,  etc.)  can 
help  people  escape  organizational  bottlenecks  and  can  rapidly  connect  people  having  with 
common  interests  and  problems.  Wiki  workspaces  can  show  networks  of  expertise  which  be  used 
as  a  platform  for  collaboration,  and  used  to  manage  processes  of  mentoring.  Peer-to-peer,  such 
networks  can  establish  an  organizational  culture  in  which  workers  can  collaborate  to  suggest 
improvements  to  business  procedures  or  policies. 

Though  this  is  all  plausible,  it  is  anecdotal  and  there  is  a  paucity  of  research  demonstrating  actual 
"learning  acceleration"  other  than  collections  of  organizational  success  stories  that  show  business 
success  (i.e.,  "the  efforts  paid  off...")  but  not  actual  learning. 

A  notion  of  accelerated  learning  can  be  found  in  the  field  of  business  management,  in  efforts 
called  High  Performance  Work  Systems  (HPWS).  A  main  idea  in  this  approach  is  to  move 
decision  making  power  down  from  higher  levels  of  management  and  administration  to  "front¬ 
line"  workers.  Reports  suggest  that  in  many  forms  of  business,  this  approach  works. 

If  tasks  are  uniform  and  stable,  and  if  there  are  well-known  solutions  to  problems 
that  arise,  then  a  traditional,  routinized,  and  centralized  work  organization  is  most 
effective.  But  if  tasks  vary  frequently  then  a  more  flexible  "polycentralized" 
organization  is  deemed  appropriate...  a  work  organization  that  makes  use  of 
workers'  imagination  and  initiative  is  superior  (Applebaum,  et  ah,  2000,  p.  36-37). 

DiBello,  Missildine  and  Struttmann  (2009)  argue  that  increasingly  in  complex  manufacturing 
sectors,  like  biotech,  greater  decision  making  responsibility  is  being  demanded  of  front  line 
workers  without  the  requisite  power  to  carry  out  these  complexity  of  the  objectives  involved.  But 
achieving  empowerment  of  front-line  workers  requires  training  to  a  higher  level  of  proficiency. 
Typical  techniques  for  empowering  front  line  workers  do  not  hold  them  accountable  to  high 
level,  non-negotiable  outcomes  because  they  focus  on  empowerment  as  an  end  in  itself,  without 
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considering  the  importance  of  empowering  workers  in  such  a  way  that  engages  their  knowledge 
and  skill  (their  intuitive  or  tacit  expertise)  to  achieve  the  higher  level  goals  of  the  organization. 

DiBello,  et  al.  demonstrated  the  use  of  scenario-based  simulations  in  training  to  instill  "intuitive 
expertise"  in  front-line  workers  in  the  biotech  industry.  The  study  represents  accelerated  learning 
in  that  realistic  scenarios  were  worked  in  compressed  time,  that  is,  durations  in  the  actual 
scenario  that  were  not  filled  by  scenario-pertinent  events  could  be  compressed  (e.g.,  bacterial 
fermentation  might  take  a  month  but  can  be  compressed  to  20  minutes). 

The  context  was  a  company  that  made  proteins  and  other  biological  materials  for  research  and 
drug  testing.  As  the  company  grew  through  acquisitions,  it  came  to  consist  of  a  number  of 
separate  teams  with  specialization  only  in  certain  areas,  reflecting  the  specializations  of  the 
individual  companies  that  had  been  acquired.  This  resulted  in  overall  inefficiency  and  a  backlog 
on  orders  that  led  to  customer  complaints  (as  much  as  $800,000  worth  of  orders  per  day  on  back 
order). 

The  researchers  studied  the  work  and  conducted  interviews  and  found  that  some  individuals 
managed  to  grow  the  right  material  in  the  needed  volumes  and  on  time.  These  individuals,  not 
highly  trained  biologists,  had  developed  proficiency  and  "intuitive  expertise"  that  was  not  being 
shared  across  the  company. 

The  simulation-based  training  was  conducted  in  a  mockup  of  the  company's  shop  floor.  In  the 
simulation-based  training,  the  researchers  developed  a  prototypical  scenario  that  unfolded 
according  to  a  normative  ideal  (such  factors  as  cost,  product  flow,  profits).  The  exercise  was 
presented  as  a  game,  in  which  teams  had  to  meet  all  customer  orders  on  time.  The  training 
utilized  a  fast-paced  operational  simulation  exercise  that  did  not  specify  the  skills  or  means  for 
greater  responsiveness;  rather  the  trainers  emphasized  only  goals  and  challenged  the  groups  to 
develop  a  solution. 

Teams  of  20  to  25  workers  ran  through  the  scenario  and  typically  applied  their  usual  approaches 
and  methods.  The  instructors  allow  the  workers  to  continue  using  their  normal  methods,  but 
increase  the  tempo  and  pressure  to  perform,  all  the  while  plotting  the  team's  performance  against 
the  ideal.  "By  the  end  of  the  first  day  the  workplace  is  in  crisis  and  the  participants  are  realizing 
that  their  resources  are  being  expended  to  react  to  mounting  problems"  (p.  21).  In  Round  Two  of 
the  exercise,  the  participants  review  their  performance  and  discus  what  went  wrong.  New 
solutions  are  generated,  and  thus  it  is  this  round  that  brings  out  the  tacit  knowledge  of  the  more 
proficient  workers,  presented  in  context  for  all  to  benefit  from.  In  a  Third  Round  the  simulation 
is  lightly  altered  so  that  it  cannot  be  worked  from  memory)  and  the  participants  try  out  their  new 
approaches. 

In  the  two-year  study  in  the  biotech  firm,  high  levels  of  performance  were  achieved  and 
sustained  by  workers.  In  the  end,  all  groups  exceeded  desired  outcomes  and  developed  skills  in 
the  process  not  normally  targeted  in  training.  They  accelerated  the  launch  of  new  products  by 
several  "game-months."  Comparing  the  two  run-throughs  of  the  exercise  that  each  team 
experienced,  even  the  best-performing  team  on  the  first  run-through  performed  worse  than  the 
worst-performing  team  on  the  second  run-through.  After  one  year,  the  novel  solutions  developed 
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in  the  exercises  had  been  put  into  place  and  a  high  performance  level  had  been  sustained.  The 
back  orders  problem  plaguing  the  company  had  been  resolved. 

This  method  has  worked  successfully  and  to  good  advantage  in  a  number  of  domain  application 
projects.  DiBello,  et  ah,  see  its  value  in  eliciting  the  tacit  knowledge  of  the  most  proficient 
workers,  calling  out  methods  and  strategies  that  do  not  work  well,  and  generating  new  and  more 
effective  methods  and  procedures. 

We  accelerate  and  intensify  the  original  process  through  which  a 
workplace  culture  of  practice  shapes  cognition  normally... 
experienced  low-level  workers  developed  approaches  that  go 
beyond  what  would  be  possible  with  simply  starting  over  with  new 
workers  or  attempting  to  "train"  them  with  best  practices  used 
elsewhere...  [Our  approach  works]  because  of  their  considerable 
content  knowledge;  they  emerge  from  our  simulation  exercises 
with  new  frameworks  but  retain  their  rich  experiential  content.  The 
new  framework  redeploys  their  rich  content  knowledge  in 
innovative  ways  (p.  29). 


2. 7.2  Accelerating  the  Acquisition  of  Teaching  Skills 

The  concept  of  accelerated  learning  has  most  recently  been  invoked  in  the  field  of  teacher 
education.  Viadero  (2010)  reports  on  a  4-year  longitudinal  study  by  the  Carnegie  Foundation, 
focusing  on  the  teaching  of  English  in  primary  schools.  In  seventeen  schools,  teachers  who 
received  additional  coaching  in  how  to  teach  literacy  became  more  effective  as  teachers.  There 
was  also  a  wide  variation  among  schools  and  among  teachers  within  schools.  Such  findings 
could  mean  acceleration  in  a  number  of  ways.  First,  it  could  be  that  many  of  teachers  were  not  so 
good  to  begin  with  (a  floor  effect).  In  fact,  "the  teachers  who  got  the  most  coaching  were  the  new 
teachers"  (p.  2).  It  could  be  that  there  was  an  acceleration  of  teacher  learning  but  that  might  be 
because  of  the  intervention  and  not  because  of  anything  substantive  about  the  training  itself  (e.g., 
there  might  be  a  Hawthorne  effect). 

2. 7.3  Acceleration  of  Perceptual  Learning 

Research  on  training  and  instructional  design  has  shown  that  advanced  learners  learn  different 
things  and  in  different  ways  than  less  advanced  learners.  One  important  research  question  is 
precisely  when  and  how  to  shift  the  level  of  the  training  materials  and  methods  (for  some  given 
domain  or  skill).  Another  question  is  whether  acceleration  can  be  achieved  by  hastening  the 
progression  from  journeyman  to  expert.  The  field  of  instructional  design  generally  focuses  on 
instruction  from  novice  levels  upwards.  Indeed,  Fadde  (2007)  notes  that  the  field  has  generally 
felt  that  training  to  high  proficiency  is  a  matter  for  mentoring  and  massed  practice,  and  not  of 
instructional  design. 

Fadde  (2007)  discussed  an  acceleration  method  that  derives  directly  from  the  research  in  the  field 
of  Expertise  Studies,  and  specifically  from  the  Recognition-Primed  Decision  Making  model 
(Klein,  1989).  Expertise  in  many  domains  is  defined  partly  in  terms  of  the  practitioner’s  ability 
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to  rapidly  perceive  patterns  that  non-experts  cannot  perceive  (Klein  &  Hoffman,  1992.). 
Furthermore,  highly  proficient  practitioners  are  often  able  to  apprehend  a  situation  and 
intuitively  know  what  course  of  action  to  take.  In  theory,  one  could  do  a  form  of  time 
compression  in  which  one  taps  into  a  corpus  of  cases  that  present  opportunities  for  perceptual 
learning,  and  thereby  “hasten  expertise”  (Fadde,  2007,  p.  1).  The  cases  do  not  all  have  to  be 
“tough”  (i.e.  emphasize  the  recognition  of  critical  patterns),  but  should  span  a  range  including 
representative  cases. 

Fadde  (2007)  notes  the  similarity  of  such  an  approach  to  methods  of  educating  attention  that 
have  been  utilized  in  sports  (e.g.,  Chamberlain  &  Coehlo,  1993;  Williams  &  Ward,  2003). 
Within  sports  science  there  is  ample  evidence  that  training  in  recognitional  skills  (e.g.,  a  baseball 
batter’s  ability  to  rapidly  recognize  the  type  of  pitch  that  was  just  thrown)  can  be  trained  using 
simulations  and  even  video,  and  that  practice  using  those  shows  positive  transfer  to  a  “live” 
recognition  task  (Farrow,  et  ah,  1998).  Part-task  recognition  training  can  be  accomplished  using 
low-fidelity  simulations.  For  instance,  skill  at  laparoscopic  surgery  can  be  trained  by  having  a 
trainee  use  a  laparoscope  to  manipulate  peas  set  atop  golf  ball  tees  (Williams  &  Ward,  2003). 
Fadde  discusses  the  specific  task  of  “temporal  occlusion.”  In  this  task  a  dynamic  stimulus  is 
suddenly  removed  from  view  and  the  trainee  then  has  to  engage  in  a  recognition,  categorization 
or  prediction  task. 

He  also  cautions  that  “...  it  is  generally  not  appropriate  to  apply  expert  schema  induction  to 
initial  learning  by  novices.  Recognition  training  does  not  replace  direct  instruction  in  rules, 
concepts,  and  procedures,  but  rather  enhances  it,  at  the  appropriate  time  in  the  learner’s 
development”  (p.  2). 

Domains  in  which  one  might  attempt  to  accelerate  the  achievement  of  proficiency  through  time 
compressed  perceptual  learning  would  include,  for  instance,  air  traffic  controllers.  In  these  and 
many  other  domains,  expertise  is  defined  in  part  by  the  ability  to  perceive  featureless  family 
resemblances  that  exist  over  multiple  and  dynamic  data  types  (Hoffman  &  Fiore,  2007).  In 
addition,  experts  have  the  ability  to  continuously  re-leam  as  the  important  patterns  change,  and 
with  advances  in  the  technology  and  displays. 


2.7.4  Demonstrations  of  Acceleration  in  Teaming 

A  training  project  that  can  be  interpreted  as  a  success  case  of  accelerated  learning,  in  the  sense  of 
improvements  to  training,  is  a  study  by  Shebilske,  et  al.  (1992)  on  part-whole  training.  This 
project  was  aimed  at  the  issue  of  how  to  train  complex  skills,  either  as  wholes  or  in  parts.  The 
studies  use  such  laboratory  tasks  as  Space  Fortress,  which  involves  such  tasks  as  joystick  control 
of  a  simulated  spacecraft,  battling  adversaries,  and  avoiding  mines.  In  one  protocol  developed 
previously,  trainees  practice  the  part  tasks  but  in  the  context  of  the  whole  task.  Periodically,  the 
part  tasks  the  trainee  is  practicing  get  changed.  This  “multiple  emphasis  on  components”  (MFC) 
protocol  has  a  positive  effect  on  learning,  compared  to  training  in  which  the  part  tasks  are 
practiced  out  of  context  (Gopher,  Weil  &  Siegel,  1989).  In  the  new  protocol  that  Shebilske  et  al. 
developed,  trainees  work  in  pairs,  with  one  trainee  practicing  half  of  the  task  while  a  second 
trainee  practices  the  other  half-task  in  the  same  whole  task.  Periodically,  the  two  trainees 
alternate  the  half-tasks.  This  Active  Interlocking  Modeling  protocol  improves  training  relative  to 
individual  practice.  Interestingly,  performance  is  linked  to  general  reasoning  ability,  suggesting 
that  each  trainee’s  opportunity  to  observer  and  reason  about  their  fellow  trainee’s  performance. 
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all  the  while  conducting  their  own  half-task,  contributes  to  the  method  effectiveness.  The  AIM 
protocol  has  been  successfully  used  in  training  by  a  commercial  airline. 


2.7.5  Intelligent  Tutoring  Systems 

Intelligent  Tutoring  Systems  (ITSs)  leverage  techniques  form  expert  systems  (e.g.,  production 
rule  systems)  and  artificial  intelligence  to  go  a  major  leap  beyond  the  first  so-called  teaching 
machines  (Forbus  &  Feltovich,  2001).  ITSs  rely  on  a  knowledge  base  of  concepts  and  rules, 
gleaned  from  subject  matter  experts.  The  computer  is  provided  with  or  infers  information  about 
the  learner’s  level  of  knowledge  and  presents  problems  that  are  appropriate  to  that  level,  and  then 
adds  progressive  difficulty,  specifically  for  the  purpose  of  pushing  the  student's  understanding  to 
a  higher  level. 

Early,  but  relevant,  developments  in  ITSs  the  SOPHIE  and  "Sherlock"  tutors  for  training  in 
troubleshooting  skills  (see  Poison,  Poison,  &  Richardson,  1988).  Sherlock  (not  an  acronym)  is  an 
ITS  for  troubleshooting  avionics  test  equipment  for  the  P-15.  The  creation  of  Sherlock  was 
premised  on  the  fact  that  the  legacy  training  focused  on:  (1)  formal  principles  rather  than  “direct 
manipulation  experiences”  that  promote  causal  reasoning  skills  (Morris  &  Rouse,  1985)  and  (2) 
the  analysis  of  individual  components  rather  than  system-level  functionalities  and  testing 
procedures  (Rouse,  1982). 

Sherlock's  production  rules  were  generated  from  a  cognitive  task  analysis  of  the  reasoning  and 
procedures  of  highly-experienced  technicians.  Sherlock  presents  to  trainees  a  series  of 
progressively  more  difficult  troubleshooting  scenarios  involving  fault  isolation.  The  training 
occurs  at  a  high-fidelity  mockup  of  the  actual  technician’s  avionics  test  station.  Coaching — 
explanations  of  an  expert’s  strategic  reasoning — was  provided  along  the  way.  The  “Sherlock” 
name  refers  to  one  of  the  icons  on  the  computer  display  that  managed  the  test  procedures:  A 
clickable  icon  showing  a  Calabash  pipe. 

Gott  (1995)  conducted  a  test  of  the  effectiveness  of  Sherlock.  Apprentices  (having  3  years 
experience)  received  the  Sherlock  training,  and  on  a  subsequent  test  they  outperformed  a  group 
of  Masters  (having  had  10  or  more  years  experience).  Apparently,  with  Sherlock  apprentices 
were  able  to  acquire  the  strategic  knowledge  of  highly  proficient  technicians.  Another  capsule 
view  of  the  Sherlock  findings  (Eesgold,  Eajoie,  Bunzo,  &  Eggan,  1992;  Eajoie,  2009  is  that 
approximately  25  hours  of  Sherlock  training  was  the  equivalent  of  4-years  of  on-the-job  training. 
Gott  attributed  the  dramatic  effect  to  the  “situatedness”  of  the  training,  the  ecological  validity  of 
the  scenarios  and  the  “learn-by-doing”  approach. 

Sherlock  presented  a  concentration  of  useful  cases  in  a  brief  period 
of  time.  The  real  world  mostly  provides  opportunities  to  do  the 
routine.  Expertise  involving  the  non-routine  is  harder  to  get  from 
everyday  work  experience  because  the  right  situations  occur  rarely 
and  often  are  handled  by  established  experts  when  they  do  occur, 
not  by  students”  (Eesgold,  2001,  p.  967). 
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Instruction  using  intelligent  tutoring  has  shown  signifieant  gains  (upwards  of  one  standard 
deviation)  over  most  other  methods  (Corbett,  et  ah,  1999).  Meta- analytic  techniques  document 
gains  of  up  to  one  standard  deviation  when  eompared  to  other  methods  as  well  as  significant 
reductions  in  time  on  training  (Dodds  &  Fleteher,  2004;  Wisher  &  Fleteher,  2004). 

One  caution,  appropriate  in  the  context  of  the  present  Report,  is  that  Sherloek  (and  many  other 
ITSs)  trains  for  fixed  tasks.  Certainly,  teaching  the  functionality  of  the  avionies  testing 
equipment  and  the  rationale  behind  fault  diagnosis  tasks  will  be  better  than  training  rote 
proeedures  with  minimal  sensemaking.  But  fixed  tasks  do  not  eharaeterize  modem  "eomplex 
cognitive  work"  systems. 

Likely  it  takes  four  years  for  the  avionies  teehnieians  to  experienee  the  variety  of  task  situations 
that  Sherloek  eould  present.  Given  that  experts  eome  to  aequire  understanding  of  rare  eases,  that 
is,  those  partieular  problems  or  deeisions  that  oecur  only  approximately  5%  of  the  time 
(Hoffman  et  al.  2007),  simulations  of  rare  eases  may  help  time  compress  the  learning  process 
(e.g.,  Hoffman  &  Fiore,  2007).  From  the  Sherloek  projeet,  we  know  that  it  is  possible  to  time 
eompress  the  experienee-feedbaek  eyele  for  aeeeleration  from  apprentiee  to  journeyman 
proficiency  levels.  Aeeeleration  beyond  that,  to  high  proficiency,  would  only  be  possible  if  the 
ITS  ineorporated  the  expert's  "tough  ease  knowledge."  Woolf,  et  al.  (2002)  suggest  that  tutors 
need  to  be  more  generative,  that  is,  ereate  instmetional  material  (problems,  hints,  help)  based  on 
student  performanee  simultaneous  to  engaging  in  student  modeling  sueh  that  it  is  able  to  assess 
the  student’s  knowledge  and  modify  instruction  accordingly. 

Fletcher  (2010)  reported  a  study  that  used  an  ITS  for  training  the  Navy's  Information  Teehnology 
rating.  Thirty-two  students  who  had  been  seleeted  for  the  "A"  Sehool  training  reeeived  a  16- 
week  program  of  instmction  that  included  two  weeks  of  training  using  a  prototype  tutor.  The 
tutor,  developed  by  DARPA,  was  designed  specifically  to  provide  "best  practices  of  human 
expert  tutors"  at  solving  problems  in  information  teehnology  troubleshooting  (Fleteher,  2010,  p. 
1).  At  the  end  of  the  "A"  Sehool  and  again  after  some  "C"  Sehool  training,  groups  of  trainees 
were  given  a  series  of  troubleshooting  problems  (spanning  a  range  of  difficulty)  in  a  virtual 
laboratory,  in  an  instrumented  laboratory  and  aboard  Navy  ships.  Their  performance  was 
eompared  to  that  of  experieneed  teehnieians  (average  of  about  7  years  experienee  with  40  C 
Sehools).  (The  experimental  and  eontrol  groups  were  statistieally  similar  in  terms  of  age  and 
Armed  Forees  Qualification  Test  scores.) 

The  experieneed  teehnieians  solved  about  85%  of  the  problems,  with  about  60%  of  the  solutions 
rated  as  exeellent.  The  students  who  had  reeeived  the  ITS  tutoring  solved  97%  of  the  problems 
with  90%  rated  excellent.  In  the  study  of  troubleshooting  onboard  a  deployed  ship,  most  of  the 
contributions  of  the  trainees  to  solving  actual  troubleshooting  problems  were  rated  by 
supervisors  as  substantial  or  essential.  The  projeet  aehieved  its  goal  of  raising  the  performanee  of 
newly  trained  teehnieians  from  a  50th  pereentile  to  a  98th  pereentile  level.  The  trainees  had 
mastered  all  the  "A"  School  content  in  two  weeks.  With  regard  to  the  more  advanced  "C"  School 
training,  the  tutor-based  training  produced  teehnieians  with  skill  comparable  to  that  of 
teehnieians  with  12  years  of  experienee.  This  is  a  elear  demonstration  of  both  rapidized  training 
and  aeeelerated  profieieney,  ineluding  a  demonstration  of  transfer  to  the  operational  eontext. 
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2.7.6  The  Army's  "Think  Like  a  Commander"  Training  Program 

A  training  strategy  that  has  in  recent  decades  received  considerable  attention  is  the  teaching  of 
critical  thinking  skills  by  presenting  trainees  with  realistic  scenarios  and  also  with  support  that 
takes  the  form  of  guidance  concerning  the  pattern  detection,  decision  making  and  reasoning  of 
experts.  Studies  of  the  effectiveness  of  this  form  of  training,  primarily  in  the  application  area  of 
military  tactical  decision  making,  have  been  generally  successful  (e.g.,  Cohen,  Freeman  & 
Thompson,  1997,  1998;  Klein,  McCloskey,  Pliske,  &  Schmitt,  1997). 

The  Think  Like  a  Commander  (TLAC)  training  program  can  be  thought  of  as  an  attempt  at 
accelerating  the  achievement  of  high  levels  of  proficiency  at  cognitive  work,  that  is,  both 
rapidized  training  and  accelerated  proficiency  (see  Shadrick,  2009).  The  Program  relies  on 
scenario-based  exercises  using  difficult  cases,  with  guidance  and  instruction  from  a  highly 
experienced  officer,  on  an  assumption  that  this  constitutes  deliberate  practice.  The  goal  is  to 
allow  officers  to  model  their  battlefield  understandings,  plans,  visualizations,  and  decisions  after 
expert  tactician's  thinking  patterns  (Shadrick  &  Lussier,  2004). 

Trainees  are  presented  with  vignettes  from  battlefield  studies,  that  is,  interviews  with 
experienced  commanders  who  retrospected  about  previously-experienced  cases.  This  is  the 
interpretation  of  accelerated  learning  as  Facilitated  Knowledge  Sharing.  Often,  military  lessons 
learned  quickly  become  lessons  forgotten  as  they  are  left  to  reside  in  libraries. 

In  the  TLAC  Program,  trainees  work  through  a  dozen  or  so  scenarios  in  simulated  compressed 
time,  making  decisions  about  deployment  of  assets,  visualizing  the  battlefield  and  coping  with 
contingencies  as  the  scenario  unfolds.  They  are  coached  in  how  experts  look  at  problems  and 
how  the  adversary  would  think  about  the  problems.  Trainees  are  encouraged  to  work  intensively 
on  the  problems,  and  thus  we  might  say  this  training  represents  highly  motivated  "practice  with 
zeal."  The  TLAC  training  can  involve  scaffolding  by  a  trainer,  or  it  can  be  conducted  by  trainees 
without  benefit  of  such  scaffolding. 

In  a  series  of  studies,  Shadrick  and  colleagues  (Shadrick,  et  ah,  2007;  Shadrick  &  Lussier,  2004, 
2009;  Shadrick,  Lussier,  &  Fultz,  2007)  evaluated  the  U.S.  Army's  "Think  Like  a  Commander 
Program"  (TLAC),  and  also  the  "Captains  in  Command  Training  Program  for  Adaptive  Thinking 
Skills"  (which  is  a  PC -based  version  of  TLAC)  (Shadrick  &  Fite,  2009).  They  demonstrated  that 
the  training  leads  to  significant  improvement  in  the  ability  to  rapidly  analyze  a  tactical  situation 
in  order  to  identify  the  critical  information  needed  for  tactical  decision-making.  They 
demonstrated  that  TLAC  training  improved  the  ability  of  officers  (in  the  Armor  Captains  Career 
Course)  to  create  better  company-level  Operations  Orders. 

The  2004  study  was  a  post-training  performance  evaluation  of  trainee  ability  to  analyze  tactical 
situations  in  battlefield  vignettes.  This  adaptive  thinking  measurement  instrument  was  validated 
in  a  second  study,  which  demonstrated  that  leaders  having  experience  in  Operation  Iraqi 
Freedom  and  Operation  Enduring  Freedom  performed  better  at  the  TLAC  vignettes  than  those 
without  such  experience.  For  those  without  such  experience,  performance  increased  with  rank. 
For  those  with  experience,  performance  of  captains  and  majors  was  about  the  same  as  that  of 
lieutenant  colonels.  In  other  words,  experience  trumped  rank.  The  authors  take  this  as  evidence 
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of  the  powerful  effect  of  deliberate  practice,  assuming  that  it  instills  the  same  capabilities  as 
(deliberate)  experience.  The  findings  also  suggest  that  training  using  the  (relatively  simple) 
vignette  materials  may  be  more  effective  and  efficient  than  experiential  training,  i.e.,  training 
using  more  elaborate  methods  of  simulation  and  virtual  reality.  Citing  the  work  of  Ericson  and 
his  colleagues  (Charness,  Krampe,  &  Mayr,  1996;  Ericsson,  1996;  Ericsson,  et  al.,  1993). 
Shadrick,  Eussier  &  Eultz,  2007)  conclude: 

The  maxim  “train  as  you  fight”  has  risen  to  such  a  level  of 
familiarity  in  the  U.S.  Army  that  the  value  of  the  notion  goes  almost 
unquestioned;  that  is,  that  the  best  training  method  is  to  ‘just  do  it’ 
in  a  realistic  environment.  Yet  studies  of  the  development  of 
expertise  clearly  indicate  that  “as  you  fight”  meaning  performing  in 
fully  realistic  simulated  battles  is  neither  the  most  effective  nor 
efficient  method  of  developing  expertise.  Such  “performances”  can 
help  a  novice  become  acquainted  with  applying  military  knowledge 
and  can  reinforce  existing  knowledge  in  an  experienced  person,  but 
will  not  in  and  of  themselves  lead  to  the  development  of  expertise. 

In  many  fields  where  expertise  has  been  systematically  studied, 
including  chess,  music,  and  sports,  development  beyond 
intermediate  level  requires  large  amounts  of  deliberate  practice  (p. 

17). 

The  implication  is  that  for  some  domains  and  types  of  tasks  (or  missions),  accelerated  learning 
might  be  achievable  using  relatively  simply  vignette  methods  as  opposed  to  more  costly 
simulation  methods. 

It  is  also  interesting  to  note  their  finding  that  the  training  can  be  effective  even  when  trainees 
work  the  vignettes  without  benefit  of  instructor  scaffolding. 

The  effectiveness  of  the  TEAC  training  approach  was  verified  by  a  laboratory  study  conducted 
by  TNO  Human  Eactors  in  The  Netherlands  (van  den  Bosch  &  Helsdingen,  2002).  They 
developed  a  number  of  scenarios  involving  ground-to-air  defense.  The  trainee  played  the  role  of 
the  battle  captain.  As  the  scenario  events  unfolded,  the  trainee  was  presented  with  ambiguous 
and  sometimes  conflicting  information,  to  allow  for  alternative  interpretations  of  events.  During 
scenario  "freezes,"  the  training  was  provided  with  information  about  what  the  experienced 
commander  might  notice  or  decide,  or  alternative  interpretations  of  situations.  Trainee 
performance  was  evaluated  by  the  training  leaders,  in  terms  of  the  results  of  the  trainee  actions  in 
the  scenario  outcome,  quality  of  the  trainee's  plans,  quality  of  arguments,  and  other  factors. 

Unlike  the  finding  of  Shadrick  and  Eite  (2009)  that  instructorless  training  was  effective,  the  TNO 
studies  demonstrated  that  participating  in  the  scenario-plus-scaffold  conditions  resulted  in 
improvement  whereas  merely  working  through  the  scenarios  without  the  scaffolding  did  not. 
Interestingly,  the  opening  instruction  (to  the  main  experimental  group)  to  take  a  "critical  thinking 
approach"  led  to  early  improvement  that  was  maintained  across  the  scenarios.  This  certainly 
suggests  that  critical  thinking  might  be  thought  of  as  a  stance  as  much  as  a  set  of  specific  or 
learned  skills.  If  critical  thinking  were  the  latter,  one  might  expect  to  see  gradual  rather  than 
rapid  and  immediate  performance  improvement.  As  the  authors  point  out,  additional  research 
with  various  additional  controls  is  certainly  in  order. 
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In  a  second  experiment  at  TNO,  one  group  of  trainers  was  itself  trained  in  the  "critical  thinking 
instructional  method"  and  its  components,  and  another  group  of  trainers  was  instructed  to  follow 
their  usual  instructional  methods.  In  this  study,  the  domain  was  anti-air  and  anti-surface  naval 
warfare  and  the  study  was  conducted  using  high  fidelity  simulations  at  the  Operational;  School 
of  the  Royal  Netherlands  Navy.  In  this  case,  the  trainees  worked  in  teams  and  team  performance 
was  evaluated  (information  exchange,  leadership,  time  management,  quality  of  plans,  and  so 
forth).  This  study  also  found  positive  benefits  from  the  critical  thinking  training,  in  this  case  with 
the  control  for  possible  biasing  on  the  part  of  the  trainers. 

The  authors  discussed  their  work  as  the  training  of  "critical  thinking"  (See  also  Cohen,  et  ah, 
1997,  1998;  Klein,  et  ah,  1997),  but  in  practice  it  seems  to  be  the  training  of  critical  thinking  in 
context  of  the  military  tactical  decision  making.  We  point  this  out  because  it  is  important  to 
separate  the  claim  of  transfer  of  training  in  "critical  thinking"  skill — reflective  and  self-critical 
assessment  of  performance —  from  the  claim  of  performance  improvement  due  to  the  training  of 
context  or  domain- specific  critical  thinking  strategies.  Nevertheless,  the  findings  from  the  TNO 
studies  certainly  justify  the  continued  study  and  expansion  of  use  of  the  Think  Like  a 
Commander  form  of  training. 

2. 7. 7  Simulation-Based  Training 

The  notion  that  simulators  could  significantly  improve  training  of  pilots  is  explicit  in  papers 
dating  back  decades,  the  reference  being  to  acceleration  in  the  sense  of  achieving  a  base  level  of 
proficiency  more  quickly  than  in  current  or  legacy  training.  The  earliest  reference  to  the  notion 
of  acceleration  (that  we  can  find)  is  in  an  Army  Report  by  Wladyslaw  Wielowski  dated  1990  and 
titled  “Trainers  (Simulators)  that  Facilitate  and  Accelerate  Learning.” 

Today,  there  is  a  consensus  of  researchers  that  serious  games,  synthetic  environments  and  virtual 
reality  trainers  can  be  used  to  very  good  effect  in  training.  Use  of  high-fidelity  simulations  of 
problem  scenarios  for  training  is  widely  recognized  as  having  promise  in  the  further 
improvement  of  training,  and  generally  with  respect  to  a  notion  that  the  progression  from 
apprentice  level  to  high  proficiency  levels  can  be  accelerated  (Dawson  &  Caird-Daley,  2009). 
Chatham  (2009a)  calls  it  the  "20th-Century  revolution  in  training." 

This  revolution  progressed  from  computer-aided  instruction  that  at  adopted  traditional  classroom 
teaching  methods  (read  and  test),  and  then  to  instruction  that  leveraged  more  of  the  capabilities 
of  computers  (games,  video  demonstrations,  etc.),  and  then  to  more  high  fidelity  games.  Most 
recently  it  has  branched  from  the  workstation  context  to  1)  the  immersive  context  of  simulators, 
2)  the  "distributed"  context  of  distance  learning  and  collaboration  using  the  internet,  and  3)  the 
immersive  context  of  large-scale  live  simulated  environments.  Chatham  (2009a)  calls  this  new 
kind  of  training  "Engagement  Simulation."  He  dates  it  to  the  1970s  and  a  serendipitous 
"experiment"  from  the  Vietnam  War,  in  which  low  air  war  kill  ratios  led  the  Navy  to  institute 
Top  Gun  training,  while  the  Air  Force  did  not  do  anything  similar.  Great  improvement  in  the 
Navy  flyers,  but  not  in  the  Air  Force,  signaled  the  efficacy  of  the  new  kind  of  training.  It  has  led 
to  more  of  this  kind  of  training,  for  example,  the  very  effective  NTC  (discussed  below). 
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Chatham  discusses  several  successes,  including  DARWARS  Ambush  and  the  Tactical  Language 
and  Culture  Trainer.  Despite  its  successes,  Chatham  cites  problems  with  this  kind  of  training, 
including  expense,  logistics,  and  time.  The  fact  that  engagement  simulation  takes  time  is 
interesting  in  context  of  the  present  Report  on  acceleration  of  learning.  The  hypothesis  would  be 
that  Engagement  Simulation  training  takes  time  in  the  sense  of  lengthening  the  period  of  pre¬ 
deployment  training,  but  arguably  it  saves  time  (and  lives)  by  getting  trainees  up  to  a  higher- 
than-traditional  level  of  proficiency  before  deployment.  That  is,  skill  is  acquired  in  training  that 
would  ordinarily  only  be  acquired  once  the  "boots  were  on  the  ground,"  and  at  risk.  Furthermore, 
engagement  simulation  can  train  in  warfighting  skills  and  situations  that  would,  in  the  actual 
deployment  only  be  experienced  opportunistically. 

Chatham  suggests  a  form  of  simulation-based  training  which  is  less  than  immersive,  that  is, 
serious  games  which  are  portable,  on-demand,  and  manipulable. 

2.7.8  Engagement  Simulation-Based  Training:  "Top  Gun" 

An  example  of  a  military  training  program  that  can  be  regarded  as  a  success  case  of  accelerated 
proficiency  is  the  U.  S.  Navy  “Top  Gun”  program  (Chatham,  2009a,b).  Dissatisfied  with  the  kill 
ratio  in  the  air  war  over  Vietnam,  the  Navy  established  a  training  program  involving  intensive 
practice  at  "dog-fighting."  Trainees  subsequently  engaged  the  enemy  at  a  12-to-l  ratio  (while  Air 
Force  performance  remained  at  the  2-to-l  level.  This  finding  was  calculated  over  100 
engagements  involving  multiple  types  of  aircraft  and  weapons  systems,  and  thus  the  results  are 
confidently  attributable  to  the  training  (Gorman,  1990).  (Data  on  World  War  II  pilot  performance 
showed  that  the  odds  of  being  shot  down  were  very  high  for  first  engagements,  but  that  the 
probability  of  being  shot  down  decreased  significantly  for  pilots  as  they  gained  experience.) 
Following  the  Top  Gun  success,  other  services  commenced  programs  of  large-scale  simulation- 
based  training. 


2.7.9  Engagement  Simulation-Based  Training:  The  Army's  National 
Training  Center 

At  the  National  Training  Center  (NTC),  there  is  a  simulation  of  a  town  in  the  Middle  East, 
populated  by  hundreds  of  actors  (many  of  them  of  Middle  Eastern  origin).  Battalions  in  training 
for  deployment  come  to  the  NTC  for  weeks-long  exercises  against  a  resident  battalion 
specifically  prepared  to  challenge  the  trainees.  Plot  threads  involving  insurgents,  mayors,  etc., 
with  the  scenario  design  based  on  anticipated  situations  the  trainee  battalion  will  experience  in 
theater.  Figures  2.3  and  2.4  present  photo  montages  of  the  NTC. 
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Figure  23.  A  photo  montage  of  the  NTC  "village"  area. 
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Figure  2.4.  A  photo  montage  of  the  NTC  operations  facilities  and  activities. 

Performance  is  not  measured  by  simple  comparisons  of  success;  trainees  are  not  judged  by  how 
well  they  performed  against  the  adversary  (the  resident  Red  Force  is  more  capable).  Rather,  they 
are  judged  in  terms  of  their  achievement  of  reasoning  strategies  and  skills.  The  NTC  has  reported 
significantly  increased  rates  of  success  post  training  (light  infantry  30:1,  combined  arms  teams 
15:1,  Regiments/brigades  5:1  success  rates). 

Gorman  (1990)  laid  out  these  key  features  of  this,  and  other  successful  Engagement  Simulation- 
based  training  programs: 

•  Training  is  given  to  teams  and  not  just  individuals, 

•  Training  uses  the  actual  systems  that  will  be  experienced  in  the  operational  setting. 

•  Training  involves  the  use  of  complete  scenarios,  run  from  start  to  finish 

•  Feedback  is  continuously  available  to  and  among  trainees,  and  is  extensive  during 
feedback  periods. 

•  Feedback  and  discussion  covers  situation  awareness,  decision  processes,  state  of  mind, 
and  other  aspects  of  the  cognitive  work. 

•  Trainees  are  encouraged  to  think  deeply  about  situations  and  problems  and  share  ideas, 
feelings,  and  insights.  Trainees  are  encouraged  to  retrospect  about  and  evaluate  their 
reasoning. 
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•  Feedback  comes  from  trainees  and  not  just  trainers. 

•  Independent  red  and  blue  forces,  with  the  aggressor  force  fully  prepared  and  capable  of 
using  actual  adversary  tactics. 

•  Red  force  warriors  are  themselves  of  high  proficiency. 

•  The  trainees  experience  tougher  situations  than  those  typical  of  what  they  are  likely  to 
experience  in  theater. 


2.7.10  Current  Department  of  Defense  Efforts  at  Rapidized  Training 

The  Air  Force  Weather  Agency  has  launched  a  program  to  train  "Forecast  Ready"  forecasters 
possessing  high-level  knowledge  beyond  intermediate  skill  levels.  The  goal  of  the  training  is  for 
forecasters  to  be  capable  of  using  mesoscale  models  (computer  models)  and  remote  sensing 
tools.  The  goal  is  for  the  forecasters  to  be  able  to  "explain  the  reasoning  behind  the  forecast" 
(McNulty,  2005,  p.  5).  This  description  suggests  that  the  target  is  the  advanced  journeyman  to 
expert  level  of  proficiency,  which  is  generally  achieved  in  forecasting  after  years  of  experience 
(see  Hoffman,  et  ah,  2000).  It  is  also  a  proficiency  that  is  climate  or  local  specific,  generally.  To 
make  it  possible  for  training  to  accelerate  learning,  great  reliance  will  be  placed  on  mechanisms 
of  distance  learning  (Pagitt,  2005).  "The  7th  Weather  Squadron  is  moving  at  lightning  speed 
towards  a  new  training  initiative...  a  premier,  just-in-time  combat  field  skills  training  course..." 
(7th  Weather  Squadron,  2005,  p.  16).  This  initiative  can  be  seen  as  following  the  NTC  model.  Its 
focus  is  on  making  sure  that  forecasters  are  tactically  and  technically  prepared  prior  to  actual 
deployment.  The  method  is  to  site  students  at  a  large  training  area  for  a  week  of  instructions 
(vehicle  maintenance,  generator  maintenance,  simulated  convoy  operations,  etc.)  and  experience 
with  command  and  control  procedures,  in  addition  to  practice  at  tactical  meteorological 
forecasting. 

Kirschenbaum,  et  al.  (2009)  describe  an  innovation  in  training  for  sonar  technicians  and  fire 
control  technicians  aboard  a  submarine.  The  goal  was  to  rapidize  the  acquisition  of  skill  during 
training.  Traditionally,  after  the  schoolhouse  and  early  in  the  first  deployment,  the  technicians 
were  still  considered  trainees,  and  spent  most  of  their  effort  at  extensive  mentored  training. 
Furthermore  the  schoolhouse  training  was  traditional  (lectures,  Powerpoints  tests).  The  newer 
method  was  based  on  an  thorough  cognitive  task  analysis  of  the  technicians'  jobs.  The  results 
supported  the  creation  of  "job-oriented  simulations"  (computer-based  instruction)  resulting  in 
accelerated  skill  development  during  training  and  increased  proficiency  level  when  the 
technician-in-training  arrived  on  the  submarine. 


2.7.11.  Conclusions  Concerning  Acceierated  Learning 


Acceleration  in  the  sense  of  rapidized  training  has  been  demonstrated:  The  quickening  of 
training  without  reduction  in  the  proficiency  level  achieved  at  the  end  of  the  training.  Indeed, 
indications  are  that  rapidization  methods  lead  to  higher  levels  of  proficiency  than  legacy  training. 

Acceleration  in  the  sense  of  achieving  higher  levels  of  proficiency  has  also  been  demonstrated, 
and  in  complex  military  jobs/tasks.  Demonstrations  of  this  also  entail  one  of  the  meanings  of 
acceleration,  the  rapidized  transposition  of  lessons  learned  into  the  training  context. 
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All  of  the  methods  used  in  training  for  rapidization  and  for  accelerated  proficiency  depend  on 
one  or  another  form  of  time  compression.  One  form  is  that  of  packing  more  varieties  of 
experiences  into  the  training  and  not  merely  shortening  the  training.  For  instance,  proficiency  in 
a  domain  may  depend  on  having  had  a  variety  of  experiences,  any  one  of  which  might  only  arise 
rarely.  Through  case  libraries,  the  training  can  "escape"  real  time.  A  second  sense  of  time 
compression  is  to  truncate  events  that  transpire  within  a  scenario.  For  example,  a  troop 
movement  during  which  little  of  consequence  happens  can  be  truncated. 


2.8.  Summary  of  Findings  From  the  Literature  Review 

Proficiency 

•  We  have  rich  descriptions  of  proficiency  and  the  development  of  proficiency,  and  robust 
methods  for  proficiency  scaling. 

•  We  know  what  makes  the  achievement  of  high  proficiency  difficult.,  and  this  knowledge 
feeds  directly  into  instructional  design. 


Training 

•  The  variables  affecting  training  (types  and  sequences  of  practice,  domain  complexity, 
learning  styles,  etc.)  all  interact  in  complex  ways,  making  generalization  difficult. 

•  Problem-Based  Learning  shows  promise  for  training  to  perform  in  dynamic  and  complex 
domains  where  tasks  are  not  fixed. 

•  We  have  rich  descriptions  of  types  of  feedback  and  know  that  feedback,  in  general,  is 
necessary  and  helpful,  but  generalizations  about  the  timing  of  feedback  are  always 
tentative  (e.g.,  immediate  feedback  may  not  contribute  to  long-term  achievement  of 
proficiency). 

•  The  goal  of  training  is  not  to  make  learning  easy  but  to  make  it  easier  and  quicker  than  it 
would  otherwise  be.  In  initial,  intermediate  and  advanced  levels  of  training,  there  must  be 
problems  that  present  "desirable  difficulties."  Training  at  advanced  levels  cannot 
oversimplify.  Indeed,  high  proficiency  is  defined  in  part  by  the  ability  to  cope  with  tough 
cases. 

•  In  general,  mentoring  is  highly  valuable  for  the  trainee  and  the  organization.  In  many 
cases  mentoring  (vs.  instruction)  is  necessary. 

•  In  complex  domains  scenario-based  training  is  highly  valuable  and  arguable  necessary 
(e.g.,  in  training  for  situation  awareness,  training  for  adaptivity). 

•  Training  should  train  for  meta-cognition.  This  is  referred  to  as  "reflective  training"  and  as 
"training  to  learn." 
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Transfer 

•  It  is  difficult  to  make  generalizations  about  transfer,  since  patterns  depend  on  the  type  of 
task.  Perceptual-motor  skill,  declarative  knowledge,  and  reasoning  skill  are  often 
conflated  in  the  literature. 

•  High  proficiency  skill  and  knowledge  in  some  domain  are  useful  in  solving  problems  in 
that  domain  and  are  of  limited  transferability.  Yet,  the  phenomena  of  expertise  prove  that 
there  must  be  transfer  of  knowledge  across  problem  types  within  a  domain. 

•  There  are  a  number  of  different  possible  kinds  of  transfer  and  we  have  rich  descriptions 
of  these. 


Retention/Decay 

•  Our  knowledge  of  decay  and  the  general  form  of  the  decay  function  is  fairly  robust. 

•  Many  interacting  factors  influence  retention/decay;  not  all  forms  of  knowledge/skill 
decay  at  the  same  rate  and  there  is  some  inconsistency  in  the  findings. 

•  Over-learning  is  the  best  predictor  of  retention. 

•  Better  methods  are  needed  for  measuring  over-learning. 

Teams 

•  The  study  of  team  cognition  in  complex  cognitive  work  is  a  relatively  new  area  for 
human  factors  research.  There  are  many  gaps  in  knowledge  and  methodology. 

•  Better  methods  are  needed  for  assessing  work  quality  in  the  new  team-based  contexts  for 
complex  cognitive  work. 


Accelerated  Training 

•  Rapidized  training  and  accelerated  proficiency  can  be  possible  through  the  use  of 
computer  games,  simulations,  and  immersions,  all  of  which  are  enhanced  through 
application  of  intelligent  tutoring  and  artificial  intelligence  capabilities. 

•  The  use  of  case-based  instructions,  realistic  tough  cases  with  focus  on  errors  and 
"desirable  difficulties." 

•  Expertise  is  the  source  for  training  materials.  This  entails  a  new  meaning  of  "accelerated 
learning,"  that  is,  quickening  of  the  translation  of  battlefield  lessons  learned  into  the 
training. 

•  For  acceleration  of  team  proficiency,  training  exercises  must  encourage  the  acquisition  of 
teamwork  skills  and  “people  skills.”  Training  must  take  into  account  individual 
differences  in  cognitive  style  and  intrinsic  motivation. 

•  For  acceleration,  training  exercises  must  tap  into  a  rich  corpus  of  tough  cases. 
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Training  for  accelerated  proficiency  must  rely  upon  meaningful,  corrective  feedback  that  is 
appropriately  timed  (neither  too  close  nor  too  distance  from  the  performance  being  evaluated). 


3.0  ABOUT  THE  WORKING  MEETING 

The  second  workshop,  titled  the  "Working  Meeting  on  Accelerated  Proficiency  and  Facilitated 
Retention,"  was  held  in  October  2009  in  Phoenix  AZ.  It  was  organized  by  Dr.  Dee  Andrews, 

71 1th  Human  Performance  Wing  Senior  Scientist,  Air  Force  Research  Laboratory,  Mesa,  AZ 

3.1.  Meeting  Focus  and  Topics 

Participants  were  presented  with  challenge  questions  noted  above  in  Section  1  of  this  Report.  In 
the  Meeting  read-ahead  these  challenge  questions  were  stated  succinctly  as: 

•  How  can  we  develop  operational  definitions  and  measures  of  proficiency  at  a  fine  grain? 

•  How  can  we  develop  methods  for  identifying  expert  mentors  and  revealing  their 
knowledge  and  strategies? 

•  How  can  we  best  design  training  to  promote  skill  retention  and  prevent  skill  decay  during 
periods  of  hiatus? 

•  How  can  we  train  for  adaptivity  and  the  need  to  cope  with  the  ever-changing  workplace 
and  changing  and  challenging  missions? 

•  How  can  we  train  for  resilience  and  the  need  to  cope  with  complexity  when  unexpected 
events  stretch  resources  and  capabilities? 

•  What  USAF  jobs  require  high  levels  of  proficiency  and  have  analogs  in  the  private 
sector? 


Core  topics  that  were  to  be  addressed  at  the  Working  Meeting  included  the  following: 


Measures  and  Levels  of  Proficiency 

This  topic  addresses  the  measurement  methodology  to  determine  levels  of  proficiency.  This 
needs  to  go  beyond  the  traditional  rough  cuts  often  used  in  organizations  and  expertise  studies 
(i.e.,  expert  versus  novice)  to  achieve  finer  discriminations  along  the  skill  continuum  (e.g., 
Hoffman,  1998).  The  challenge  is  to  establish  reliable  and  valid  markers  that  might  be  used  for 
such  things  as  placement,  promotions,  and  for  use  in  assigning  practitioners  to  skill  levels  in 
research  studies.  In  this  regard,  measurement  of  performance  for  all  complex  sociotec hnical 
systems  must  also  go  well  beyond  traditional  measures  of  human  performance  (Alberts  &  Hayes, 
2003). 

Selection  and  Scaffolding  for  the  Development  of  High  Proficiency 

Traditional  hiring  practices  have  not  served  well  to  predict  who  will  eventually  be  a  high 
performer.  New  methods  need  to  be  investigated.  For  instance,  it  may  be  that  common  screening 
tools  at  hiring  will  need  to  be  replaced  or  augmented  by  various  indicators  that  only  become 
apparent  after  some  period  of  time  functioning  on  the  job.  Along  with  identification,  there  will 
need  to  be  programs  for  supporting  progressive  development  beyond  what  are  typically  taken  to 
be  "professional  development"  activities  in  many  organizations. 
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Mentoring  and  Apprenticeship 

Many  aspects  of  work  are  best  learned  "at  the  bench,"  under  guidance  from  a  skilled  mentor. 
These  include  what  is  often  called  "tacit  knowledge,"  that  is,  many  aspects  of  "know  how," 
where  opportunities  for  learning  largely  manifest  themselves  in  the  course  of  real,  active  work,  in 
the  social  context  of  work  (Schon,  1983).  We  need  to  be  able  to  identify  excellent  mentors,  study 
them  to  reveal  the  knowledge  and  strategies  of  mentors,  and  develop  programs  for  training 
mentors.  We  also  need  to  determine  what  job  practices  are  best  suited  for  mentoring,  as  opposed 
to  using  other  kinds  of  training  techniques. 

Learning  for  Flexibility,  Adaptability,  and  Resilience 

Robustness  is  the  ability  to  maintain  effectiveness  across  a  range  of  tasks,  situations,  and 
conditions.  Resilience  is  the  ability  to  recover  from  a  destabilizing  perturbation  in  the  work  as  it 
attempts  to  reach  its  primary  goals.  Adaptivity  is  the  ability  to  employ  multiple  ways  to  succeed 
and  the  capacity  to  move  seamlessly  among  them.  Fundamental  to  the  achievement  of 
robustness,  resilience  and  adaptivity  is  the  opportunity  to  practice  at  problems  that  stretch  current 
competency  (Ericsson  &  Lehmann,  1996;  Feltovich,  et  ah,  1997).  However,  modern  work 
presents  challenges  for  achieving  such  adaptive  skills.  For  much  of  history,  individuals  have 
developed  mastery  and  resilience  by  focusing  intensely  on  a  relatively  circumscribed  task  or 
suite  of  skills,  e.g.,  chess  or  welding.  Modern  work  (especially  in  sociotechnical  work  systems) 
has  become  much  more  unstable  and  complex.  People  change  jobs,  jobs  change,  orders  change, 
technology  intrudes  and  changes  the  work,  customization  in  products  and  services  requires 
versatility  beyond  the  assembly  line,  etc.  Can  we  train  (or  select  for)  expertise  in  flexibility  and 
resourcefulness?  What  kind(s)  of  learning  and  materials  does  it  require?  For  example,  one 
method  that  has  been  used  is  to  have  workers  rotate  through  other  job  roles  that  are  integrated 
with  their  own  work. 

Addressing  All  of  These  Factors  in  the  Context  of  Teams 

In  addition  to  fluidity,  customization,  etc.,  another  prominent  feature  of  modern  work  involves 
teams  and  teamwork.  For  functional,  workable  acceleration  to  be  possible,  training  exercises 
must  encourage  the  acquisition  of  teamwork  skills.  An  expert  group  is  not  the  same  as  a  group  of 
experts  (Salas,  et  ah,  2006).  While  many  skills  may  be  in  common,  teamwork  engages  others,  for 
example,  certain  kinds  of  "people  skills,"  coordination  skills,  understanding  of  others  and  their 
"ways,"  and  the  ability  to  develop  and  maintain  satisfactory  mutual  understanding,  "common 
ground"  (Klein,  Feltovich,  Bradshaw,  &  Woods,  2004). 

Identify  Candidate  Domains  for  Study 

What  domains  might  be  a  focus  for  attempts  to  accelerate  the  achievement  of  expertise  or  assess 
methods  for  facilitating  retention?  We  seek  domains  that  are  interestingly  different  and  yet  share 
these  features: 

•  Are  domains  within  the  military  that  are  of  high  complexity,  are  mission  critical,  and 
require  high  levels  of  proficiency,  and  involve  decision  making  under  uncertainty. 

•  Are  domains  that  rely  on  extensive  knowledge  and  flexible  reasoning  (not  just  “skill”) 
since  the  cognitive  work  is  sensemaking  of  complex,  dynamic  and  nonlinear  patterns  of 
causation, 

•  Have  an  analog  in  the  private  or  public  sector. 
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•  Are  hard  to  learn  and  do  well,  as  attested  to,  for  instance,  by  high  wash-out  rates  and  few 
extant  experts, 

•  Have  in  the  past  required  very  long  training  and  on-the-job  experience. 

For  all  of  the  topics,  the  participants  were  encouraged  to  list  opportunities,  obstacles,  and 
research  requirements  for  the  acceleration  of  proficiency  and  facilitation  of  retention. 

3.2.  Roster  and  Agenda 

The  roster  of  participants  was  shaped  by  consideration  of  the  many  disciplinary  paradigms  that 
pertain  to  the  concepts  of  accelerated  learning.  Input  would  be  needed  from  specialists  in 
training,  instructional  design,  human  resource  management,  and  longitudinal  research  design. 
The  Working  Meeting  would  need  to  have  participation  by  experimental  psychologists, 
organizational  sociologists  and  educational  psychologists.  The  Meeting  would  need  input  by 
researchers  whose  work  has  focused  specifically  on  one  or  more  of  the  following  topic  areas: 
knowledge  elicitation,  the  development  of  proficiency,  learning  retention  and  decay,  case-based 
reasoning  and  instruction,  and  mentoring/tutoring. 

The  Participant  Roster  and  the  meeting  Agenda  are  presented  in  Appendix  A.  The  formats  of  the 
Meeting  activities  were  Background  setting  presentations,  whole-group  presentations,  and 
Break-Out  Group  sessions  that  focused  on  one  or  more  of  the  core  topics  and  challenges. 

The  group  photograph  is  shown  in  Figure  3.1,  and  the  legend  on  the  following  page. 
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Figure  3.1.  The  group  photograph. 
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Group  Photo  Legend 


Back  Row 

R.  Hoffman,  K.  Gluck,  S.  Fiore,  D.  Andrews,  D.  Ziebell,  A.  Gibbons 
Middle  Row 

F.  Greitzer,  M.  Krusmark,  K.  Addis,  K.  VanLehn,  L.  DiBello,  G.  Klein,  R.  Patterson,  R.  Proctor, 
P.  Feltovich,  H.  O'Neil,  L.  Magee,  W.  Bennett  (there's  a  bit  of  staggering  in  this  row) 

Front  Row 

T.  Branaghan,  R.  Hubal,  B.  Pierce,  P.  Fitzgerald,  Col.  Walker,  S.  Goldberg,  J.  Freeman,  A. 
Healy,  Lt  Col  Hilger,  J.  Bradshaw.  N.  Charness,  K.  Velkey 

Not  Pictured 

Herb  Bell,  Rand  Spiro,  Maj.  Brian  Healy,  Alex  Lincoln 
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3.3  Pre-Meeting  Activities 

White  Paper  material  was  composed  and  shared  with  all  of  the  participants  prior  to  the  meeting. 
That  document  is  presented  here  in  Appendix  B.  Along  with  it  was  sent: 

Hoffman,  R.  R.,  Fiore,  S.  M.,  Klein,  G.,  Feltovich,  P.  &  Ziebell,  D.  (2009,  March/ April). 
Accelerated  learning  (?).  IEEE:  Intelligent  Systems,  pp.  18-22. 

Additional  read-ahead  papers  were  made  available  at  the  meeting  web  site.  These  were: 


DiBello,  L.,  Missildine,  W.,  &  Struttman,  M.  (2009).  “Intuitive  Expertise  And  Empowerment: 
The  Eong  Term  Impact  Of  Simulation  Training  On  Changing  Accountabilities  In  A  Biotech 
Eirm.”  Report,  WTRI  Incorporated. 

Eeltovich,  P.  J.,  Coulson,  R.  E.,  Spiro,  R.  J.,  &  Dawson-Saunders,  B.K.  (1992).  Knowledge 
application  and  transfer  for  complex  tasks  in  ill-structured  domains:  Implications  for  instruction 
and  testing  in  biomedicine.  In  D.A.  Evans  and  V.E.  Patel  (Eds.).  Advanced  Models  of  Cognition 
for  Medical  Training  and  Practice.  Berlin:  Springer- Verlag. 

Kalyuga,  S.,  Ayres,  P.,  Chandler,  P.,  &  Sweller,  J.  (2003).  The  expertise  reversal  effect. 
Educational  Psychologist,  38,  23-31. 

Koedinger,  K.  R.,  Pavlik,  P.,  McEaren,  B.  M.,  Aleven,  V.  (2009).  Is  it  Better  to  Give  than  to 
Receive?  The  Assistance  Dilemma  as  a  Eundamental  Unsolved  Problem  in  the  Cognitive  Science 
of  Eeaming  and  Instruction. 

Eajoie,  S.  P.  (2003).  Transitions  and  trajectories  for  studies  of  expertise.  Educational 
Researcher,  32,  21-25. 

Townsend,  M.  (2009/ April).  Eeveraging  human  networks  to  accelerate  learning.  Chief  Learning 
Officer,  pp.  34-37. _ 


In  addition  to  providing  their  biographical  sketches  (see  Appendix  C),  participants  were  asked  to 
submit  Pre-Meeting  Position  papers  (see  Appendix  D). 
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4.0  INTEGRATION  OF  THE  WORKING  MEETING  PRESENTATIONS 
AND  DISCUSSIONS 

Looking  across  the  Position  Papers  (Appendix  D),  the  Presentations  (Appendix  E),  the 
discussions  based  on  the  Presentations,  and  the  Break-Out  Group  discussions,  this  variety  of 
topics  was  discussed: 

•  Domains  of  Study 

•  Some  Paradoxes  and  Ironies  of  Training 

•  Research  Challenges 

•  Methods  and  Designs  for  Acceleration 

4.1  Domains  of  Study 

Many  differences  and  similarities  can  be  seen  comparing  professional  domains.  For  example, 
with  regard  to  technology  important  dimensions  of  difference  include  the  newness  of  the 
technology  (e.g.,  new  software  versus  legacy  systems)  and  the  extent  to  which  workplaces  and 
jobs  depend  on  information  technology.  In  some  specializations,  information  technology  is  not 
the  critical  aspect  of  the  job  whereas  in  others  it  is.  In  his  analysis  of  system  administrators, 
Haber  (2005)  noted  that  IT  systems  often  consist  of  a  multitude  of  components  ranging  from 
“HTTP  servers,  web  application  servers,  authentication  servers,  content  servers,  database 
management  systems,  network  load  balancers...  distributed  across  multiple  networks  and 
multiple  operating  system  platforms,  and  each  of  the  components  may  have  hundreds  or 
thousands  of  configuration  parameters”  (p.  1).  Thus,  in  the  course  of  troubleshooting  these 
systems,  one  needs  to  both  comprehend  how  all  of  these  varied  components  interact  and  to 
integrate  system  information  into  a  single  picture. 

In  some  domains,  such  as  firefighting  and  many  military  subdomains,  experts  who  confront 
problem  situations  sometimes  cannot  engage  in  deliberative  analysis  of  options.  In  emerging, 
high-stakes,  time -pressured  situations  there  is  no  time  for  rational  analysis  of  options,  costs, 
risks.  Normative  decision  theory  argues  that  problem  solving  can  only  be  optimized  if  a  rational 
analysis  is  conducted.  But  such  analysis  is  precluded  in  time-pressured  situations.  Studies  of  fire 
fighters,  nurses,  managers,  and  other  domain  experts  have  shown  that  experts  more  typically 
engage  in  recognition-primed  decision  making  (Klein,  1989).  In  this  process,  an  inspection  of 
information  leads  directly  and  immediately  to  a  path  of  action,  and  typically  the  first  option 
selected  by  the  expert  is  the  optimal  or  best  one. 

In  some  specializations  the  work  involves  high-level  cognition  but  also  involves  tough  physical 
work  (e.g.,  turbine  maintenance).  A  few  particular  domains  were  called  out  for  discussion  and 
analysis  at  the  Working  Meeting. 

Second  Language  Learning  as  a  Domain  Case  Study 


Learning  a  second  language  to  a  point  of  high  proficiency  (i.e.,  fluent  conversation  on  a  dialect) 
has  been  of  immediate  important  to  the  U.  S.  military.  Circa  Operation  Desert  Storm  and  just 
thereafter,  it  was  initially  extremely  difficult  for  the  U.  S.  military  to  meet  the  demands  for  fluent 
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translators/interpreters,  especially  for  Category  II  (security-cleared)  translators  (salary  over 
$200,000).  After  meeting  goals  of  recruiting,  and  subsequent  deployment,  significant  numbers  of 
linguists  quit.  Language-fluent  interpreters  that  the  U.  S.  military  has  sent  to  Middle  East  tend  to 
be  older  individuals  not  prepared  for  the  exertion  or  stress  (Straziuso,  2009).  An  additional 
problem  is  that  the  diversity  of  languages  and  dialects  means  that  many  people  who  are  fluent, 
are  fluent  only  in  one  or  dialect. 

Training  in  this  domain  was  discussed  in  a  presentation  at  the  2008  Accelerated  Learning 
Workshop,  by  Dr.  Ray  Clifford.  The  higher  levels  of  proficiency  are  defined  by  the  ability  to 
speak  colloquially  and  conversationally  so  as  to  “pass”  for  a  native  speaker,  and  at  the  highest 
level,  the  ability  to  think  creatively  in  the  second  language.  These  two  highest  levels  of 
proficiency  are  achieved  only  rarely  by  individuals  whose  upbringing  was  monolingual.  Those 
who  do  achieve  these  levels  do  so  only  after  significant  time  living  in  the  second  language  and 
its  cultures.  What  this  suggests  is  that  here  is  at  least  one  domain  that  is  already  “accelerated”  to 
the  extent  possible.  By  this  we  mean  that  the  apparent  limits  of  selection  and  training  seem  to 
have  been  approximated. 

Electric  Utilities  as  a  Domain  Case  Study 

The  traditional  industry  preference  is  to  engineer  out  the  tacit  knowledge.  If  they  cannot  do  that, 
they  would  want  to  outsource  it.  Last  thing  they  prefer  is  to  capture,  preserve,  and  train  it.  This  is 
changing  now  to  an  emphasis  on  knowledge  capture  and  mentoring.  The  median  age  of  the 
workforce,  across  all  of  North  America,  is  50  years.  It  was  estimated  that  65  percent  of  the  senior 
engineers  are  or  will  soon  be  eligible  for  retirement.  Thus,  in  a  number  of  utilities  there  are 
knowledge  management  efforts  aimed  primarily  at  capturing  expert  knowledge  and  reasoning 
strategies,  so  that  they  might  be  integrated  into  training  for  the  next  generation.  The  knowledge 
management  effort  includes  an  increased  emphasis  on  mentoring  and  also  the  need  for 
companies  to  have  on  staff  their  own  trained  knowledge  elicitors. 

Electric  utilities  are  slow-moving  fixed  organizations  within  which  expertise  shows  itself.  Eor 
instance,  an  individual  might  begin  a  career  in  substation  and  transformer  repair,  move  up  to  a 
workforce  management  position,  and  have  years  of  getting  to  know  the  company  from  top  to 
bottom,  represent  the  company  in  its  relations  with  a  Public  Services  Commission.  An  engineer 
might  begin  a  career  in  turbine  maintenance  and  progress  to  a  planning  function,  supervising 
plant  shut  downs  and  turbine  deconstruction.  Both  of  these  examples  are  of  mission-critical  jobs. 
(Indeed,  the  shut  down  of  a  turbine  costs  the  company  millions  of  dollars  a  day!)  Eor  this  reason, 
a  notion  of  the  “super  expert”  has  emerged  in  discussions  of  the  utilities  workforce  (Ziebell, 
2009). 

The  Electric  Power  Research  Institute  (EPRI)  was  founded  in  1973.  All  the  utilities  are  members, 
as  are  some  utilities  in  other  nations.  International  as  well.  EPRI  is  a  technology  accelerator.  In 
2000  it  began  a  process  in  collaboration  with  the  Institute  for  Human  and  Machine  Cognition,  to 
identify  methods  to  capture  undocumented  worker  knowledge.  Guidelines  were  produced  in 
2002  and  2004  for  capturing  and  using  high-value  undocumented  knowledge  in  the  nuclear 
industry. 
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In  2008  EPRI  began  to  investigate  the  notion  of  accelerating  the  achievement  of  mission  critical 
expertise.  It  is  easy  to  acquire  and  instill  “middle  of  the  bell  curve”  knowledge.  But  at  an  electric 
plant  there  will  be,  say  five  out  of  1500  people  who  are  irreplaceable.  Proficiency  spans  many 
critical  jobs,  and  not  just  engineering  jobs,  examples  being  succession  planning  and  conflict 
negotiation. 

Transfer  of  knowledge  might  use  well-understood  methods,  but  still  requires  effort  if  the  need  is 
to  transfer  high-end  technical  wisdom  to  a  few  specified  individuals  whose  value  lies  at  the 
fringes  of  the  discipline  but  is  nevertheless  mission-critical  (e.g.,  individuals  who  keep  the 
company  out  of  the  newspaper;  individuals  who  make  sure  the  lights  do  not  go  out  at  Giant 
stadium  on  Monday  night).  The  high-end  experts  sometimes  change  the  mission  and  thereby 
make  the  total  organization  resilient.  There  is  the  case  of  a  fire  in  a  relay  station  building  that  had 
lots  of  asbestos.  The  expert’s  novel  solution  was  seal  off  the  building,  flush  the  insides  into  one 
single  storm  drain  and  grab  the  debris  from  there.  The  process  took  a  couple  of  weeks  (vs.  many 
weeks),  it  protected  the  public  and  the  environment,  and  kept  the  incident  out  of  the  newspapers. 
It  takes  25-35  years  to  achieve  high-end  proficiency,  related  in  part  to  the  complexity  of  the 
domain  and  the  longevity  of  the  technology.  (Only  recently  have  we  begun  the  change  from 
analog  controls  to  digital  controls.)  Many  stories  are  told  of  the  engineering  challenges  in  New 
York  City.  Engineers  have  to  maintain  and  integrate  numerous  subsystems  built  in  different  eras. 
A  senior  relay  system  protection  engineer  retired,  and  was  replaced  by  four  engineers,  each 
having  had  ten  years  of  experience.  That  did  not  suffice.  When  a  car  hit  a  pole  and  two  lines 
tripped,  both  lines  were  lost.  Across  the  two  lines  there  were  dozens  of  different  kinds  of  relays 
and  dozens  of  different  kinds  of  relay  protection  schemes,  some  50  years  old.  The  retired 
engineer  had  to  be  brought  back  in  to  fix  this  emergency  situation. 

Such  stories  make  clear  the  fact  that  a  great  deal  of  expertise  can  be  explained  in  terms  of 
Recognition-Primed  Decision  making.  High  levels  of  proficiency  can  be  defined  largely  in  terms 
of  the  ability  to  notice  complex  and  dynamic  patterns  that  others  cannot  apprehend. 


U.  S.  Air  Eorce  Pilot  Training  as  a  Domain  Case  Study 

In  the  domain  of  fighter  piloting,  there  are  no  measures  for  expertise  per  se,  but  objective 
measures  that  are  interpreted  to  reflect  proficiency  include  weapons  employment  zones,  time 
from  target  detect  to  next  action,  time  in  enemy  threat  range,  frequency  of  mode  changes  for 
sensor  systems,  frequency  of  communication,  target  of  communication,  etc..  Additional 
measures  that  are  likely  correlated  to  proficiency  are  number  of  flying  hours,  time  in  training  and 
frequency  of  training  events,  and  peer  or  supervisor  ratings  of  performance.  The  typical 
definition  of  combat  mission  ready  proficiency  is  related  to  the  number  of  flying  hours.  The 
USAE  standard  is  500  hours  of  tactical  flying.  However,  this  metric  has  been  suspect  for  some 
time  since  the  Air  Eorce  has  been  flying  a  variety  of  mission  types  in  the  past  15-20  years  where 
it  is  quite  easy  to  have  500  hours,  but  those  hours  were  obtained  flying  patrols  (limited  to  no 
tactical  employment)  over  northern  or  southern  Iraq.  So  this  variability  results  in  lots  of  fighter 
pilots  having  500,  1000,  or  more  hours  but  not  of  the  “kind”  that  would  be  indicative  of  mission 
ready  performance. 

School/training  is  regarded  as  the  pilot’s  pedigree,  and  by  analogy,  weapons  school  would  be 
“graduate”  school  or  doctoral  level  tactical  training.  At  any  time,  the  cohort  of  duty-ready  pilots 
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includes  a  mix  of  professional  and  ad-hoc  training  and  experience.  In  the  opinion  of  some, 
current  training  methods  are  based  on  a  model  dating  to  the  1980s,  and  are  becoming  out  of  date. 

The  “lost  apprenticeship”  is  a  current  concern — there  fewer  opportunities  to  develop  expertise. 
As  the  fighter  technology  becomes  more  complex,  with  the  intent  of  making  aircraft  easier  to  fly, 
the  training  requirements  can  outpace  the  capacity/opportunities  to  train.  Thus,  training  to 
gain/maintain  readiness  is  a  growing  concern.  Aging  systems  and  a  high  operational  tempo  are 
limiting  live  training  opportunities.  Range  limitations  are  reducing  the  fidelity  of  live  training. 
These  all  stand  in  conflict  with  the  increasing  costs  and  complexity  of  missions,  which  require 
efficiency  and  proficiency.  (It  costs  $35,000/hr  for  an  F-22  flight.) 

Might  there  be  mixes  of  approaches  and  technologies  for  training  that  impact  the  achievement  of 
proficiency  and  also  promote  retention  of  skill  and  expertise?  Training  methods  are  needed  that 
are  adaptive  to  learner  needs  (given  the  diversity  of  pilot  backgrounds  and  levels  of  proficiency). 
Training  needs  to  be  “rich”  in  presenting  experiences  representative  of  the  operational  context. 
Training  needs  to  be  readily  accessible  to  warfighters.  Training  should  be  seamlessly  integrated 
with  live  systems  for  continuous  learning.  Knowledge  needs  to  be  warehoused  and  re-used  in 
training  (e.g.,  in  the  form  of  managed  scenarios). 

The  problem  of  skill  decay  due  to  hiatus  is  very  salient  within  the  piloting  domain.  Generally,  it 
is  assumed  that  for  some  given  length  of  hiatus  some  particular  refresher  course  is  needed.  It  is 
indubitably  the  case  that  practice  at  flying  is  critical  post  hiatus  for  the  pilot  to  re- accommodate 
to  the  dynamics  of  flight. 

The  actual  progress  of  decay/retention  of  piloting  or  air  combat  skill  has  not  been  studied 
empirically  “in  the  wild.”  It  would  be  a  significant  challenge  to  study  air  combat  skill  decay 
under  real  world  conditions.  Such  study  would  entail  a  longitudinal  experiment  study  with 
operational  pilots,  a  manipulated  retention  interval  (say,  3  or  6  months  and  1  or  2  years).  Post 
hiatus  there  would  be  some  training  (e.g.,  5  days)  and  then  a  scenario  exercise  having  a  known 
complexity  score.  Objective  measures  would  include  mission  outcomes,  strikers  to  target,  enemy 
aircraft  killed,  friendly  mortalities,  weapon  hit  ratios,  etc.  Only  through  such  a  study  could  there 
be  an  empirical  anchor  for  our  understanding  of  skill  decay,  skill  reacquisition  rate,  and 
moderators  of  skill  decay. 

An  additional,  or  alternative,  approach  is  to  develop  and  deploy  richer  measures  of  process  skill 
that  are  applied  in  the  analysis  of  operational  performance.  This  might  provide  an  empirical 
anchor  on  levels  of  proficiency  (and  indirectly  guide  the  management  of  post-hiatus  training 
regimes).  Process/skill  measures  might  include  time  in  vulnerability  regions,  weapons 
employment  measures,  measures  of  communication  and  coordination  efficiency  and  effectivity. 

Bennett  (2009)  reported  on  a  study  of  the  decay/retention  of  performance  at  distributed  missions 
(i.e.,  joint  forces)  on  the  part  of  US  Air  Force  F-16  fighter  pilots.  The  study  involved  running 
teams  of  pilots  in  missions  using  a  high-fidelity  F-16  test  bed.  Distributed  missions,  and  the 
simulator-based  training  for  them,  involve  having  teams,  and  not  just  individuals  in  each 
simulator  run.  The  researchers  examined  initial  training  gains,  skill  decay,  skill  reacquisition 
rate,  and  moderators  of  skill  decay.  Variables  of  the  mission  scenarios  included  time  of  day, 
speed  (sub-  versus  supersonic),  range  of  adversary  threats,  force  ration,  operating  range 
restrictions,  and  others.  The  69  participating  pilots  included  ranks  from  Lieutenant  to  Major,  and 
so  had  ranges  of  age  about  26-32  years),  years  of  service  (9-10  years)  and  flying  hours  (abut  240- 
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1900  hours).  Participants  were  assigned  to  a  stable  team  and  to  one  of  two  groups  of  teams, 
which  would  experience  different  retention  intervals  (3  or  6  months). 

First,  the  teams  of  pilots  experienced  practice  missions  for  one  hour  in  a  simulator,  followed  by 
five  days  of  curriculum  including  simulator-based  team  training.  The  purpose  of  this  initial  "pre¬ 
post"  phase  was  to  gauge  the  benefit  of  training.  Then  there  followed  a  retention  interval  of 
either  three  or  six  months,  after  which  they  were  tested  in  simulated  missions  on  each  of  two 
successive  days.  Measures  were  taken  of  both  process  and  outcome.  Outcome  measures  included 
strikers  to  target,  enemy  aircraft  killed,  friendly  mortalities,  and  weapon  hit  ratios.  Process 
measures  included  communications  overlap  (a  sign  of  coordination  inefficiency)  and  speed  of  the 
fighter  plane  at  time  of  missile  launch.  (It  is  preferable  to  launch  when  the  jet  is  traveling  faster 
than  Mach-1,  so  as  to  add  momentum  to  the  missile.) 

Results  for  some  of  the  measured  dependent  variables  showed  that  the  training  made  a  difference 
in  improving  team  performance,  and  that  prior  experience  made  a  difference  (the  teams  that 
included  more  experienced  pilots  showed  less  decay).  The  length  of  the  retention  interval  also 
had  a  noticeable  impact  for  some  of  the  measures,  representing  a  slight  but  detectable  loss  of  the 
performance  gain  attributable  to  the  training. 

The  results  were  mixed  but  suggestive,  but  the  study  is  of  practical  significance  because  it  makes 
clear  the  pragmatic  difficulties  of  research  into  these  topics,  as  Bennett  pointed  out.  Statistical 
power  is  an  issue  (sample  size,  possible  ceiling  effects),  some  of  the  pilots  may  have  known 
about  the  benchmarks  for  the  types  of  missions  they  were  going  to  fly,  there  are  complex 
interactions  of  individual  performance  in  determining  team  performance,  and  there  is  no  way  to 
control  for  relevant  experiences  during  the  hiatus  period.  There  are  many  practical  issues  and 
many  research  complications  in  the  study  of  retention/decay  in  “real  world”  domains  of 
professional  practice.  The  study  was  planned  to  take  18  months,  but  ended  up  being  three  full 
years  of  work.  This  was  due  to  limited  availability  of  the  teams,  loss  of  team  members  due  to 
reassignment. 

There  are  many  practical  issues  and  many  research  complications  in  determining  best  methods 
for  training  during  hiatus  to  mitigate  decay  and  for  refresher  training  post  hiatus  to  insure  a  fast 
re- achievement  of  high  proficiency.  What  day-to-day  practice  can  you  do  during  hiatus?  While 
on  hiatus,  pilots  do  continuous  training  so  the  hiatus  is  not  purely  "no  flying."  But  time  is 
limited.  Hiatus  periods  involve  assignments  that  are  themselves  time  and  effort  intensive,  and 
there  is  only  so  much  time  during  the  day. 

There  is  also  the  practical  issue  called  “Frankenjet.”  At  the  end  of  a  hiatus  of  say  two  years,  the 
jets  being  flown  will  have  changed  avionics,  weapons,  and  other  systems.  This  raises  the  idea 
that  hiatus  training  should  focus  on  “what  is  currently  changing  in  the  operational  context.” 


UAV  Piloting  as  a  Domain  Case  Study 

The  use  of  unmanned  aerial  vehicles  (UAVs)  is  growing  in  the  military,  federal,  and  civilian 
domains.  Worldwide,  in  the  next  8  years  UAVs  will  be  a  $15.7  billion  industry.  In  the  United 
States  the  plan  is  to  acquire  9,420  mini  UAVs,  775  tactical  UAVs,  570  minimum  altitude,  long- 
endurance  UAVs,  and  60  Global  Hawks...  From  January  to  October  2007,  the  number  of  hours 
for  mission  sorties  doubled  in  the  Air  Force,  creating  a  staffing  crisis  (Donmez,  Cummings  & 
Graham,  2009,  p,  718).  The  UAV  operator  career  track  will  no  longer  be  one  defined  by 
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retraining  of  pilots.  Instead,  it  will  be  its  own  trained  specialty  and  training  programs  are  just 
now  being  instituted.  The  UAV  program  at  the  University  of  North  Dakota  has  expressed  interest 
in  collaborating  on  projects  that  might  incorporate  accelerated  learning  notions,  just  as  the  core 
training  program  is  being  designed  and  instituted.  It  is  also  note  worthy  that  UAV  operations  will 
likely  expand,  and  significantly  so,  into  the  private  sector. 

Working  Meeting  Analysis  of  Domains 

The  military  has  need  for  more  individuals  who  can  work  at  very  high  levels  of  proficiency  in  all 
of  the  areas  designated  by  COIN  (counterinsurgency),  DIME  (diplomatic,  informational,  military 
and  economic)  and  PIMESII  (political,  military,  economic,  social,  information  and 
infrastructure). 

In  1994,  Dave  Haut  (Chief  of  the  Analysis  Division  for  the 
Department  of  Defense's  Pacific  Command,)  asked  the  question, 

"Suppose  there  are  problems  in  country  X  and  the  ambassador  has 
the  choice  of  having  a  carrier  battle  group  sail  down  the  country's 
coastline  in  a  show  of  force  or  playing  golf  with  the  country's  prime 
minister;  how  does  he  decide  which  will  be  more  effective?"  The 
combat  models  of  the  time  had  no  way  of  framing  such  a  situation, 
much  less  any  hope  of  answering  the  question.... As  Bosnia, 

Afghanistan,  Haiti,  Iraq,  Somalia  and  Darfur  in  Sudan  have  shown, 

21st  century  conflicts  are  different  from  those  of  the  early  and  mid- 
20th  century.  We  don't  fight  them  in  the  same  way,  and  in  some 
cases  we  don't  fight  at  all.  Military  force  is  only  one  of  the  elements 
of  persuasion  that  are  useful  in  addressing  21st  century  conflicts. 

Eurther,  we  can't  measure  results  in  the  same  way.  Eost  lives  and 
damaged  equipment  are  part  of  the  story,  but  not  all  of  it.  To 
understand  success,  or  lack  thereof,  we  need  variables  that  measure 
(almost)  everything"  (Hartley,  2007,  p.  1). 

In  the  Working  Meeting  Break-Out  Groups,  the  most  frequently  mentioned  consideration  was 
the  pace  of  change  in  domains  of  professions  practice,  including  in  military  jobs  and  specialties. 
A  second  frequently  mentioned  consideration  had  to  do  with  teams,  and  specifically,  the 
situation  in  which  team  turnover  rates  dictate  against  the  need  to  generate  expert  teams. 

Turnover  is  different  in  different  domains  in  terms  of  teams.  In  many  military  jobs  and  missions, 
military  people  do  not  stay  together  in  teams.  It  is  hard  to  keep  people  in  military  together  as 
people  get  transferred.  War  has  its  own  effects  that  work  against  the  notion  of  building  expert 
teams.  A  central  idea  on  the  study  of  team  cognition  is  that  a  team  of  experts  is  not  necessarily  an 
expert  team,  and  that  training  for  high  levels  of  team  proficiency  involves  training  in  team  tasks 
in  addition  to  training  in  the  teamwork.  Individuals  need  skill  at  contributing  to  the  rapid 
transition  from  the  status  of  "new  team"  to  the  status  of  "high  functioning  team." 

The  challenge  for  the  Working  Meeting  participants  was  to  identify  more  than  one  candidate 
domain  for  application  and  testing  of  methods  of  accelerated  proficiency  and  facilitated 
retention.  The  domains  must  be  interestingly  different  and  must  have  military  and  civilian 
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analogs.  Clearly,  domain  choice  will  hinge  on  a  number  of  factors,  not  the  least  of  which  will  be 
need.  One  will  have  to  determine  the  needs,  within  a  domain  or  subdomain  for  a  workforce  at 
each  of  the  proficiency  levels  spanning  junior  apprentice  to  senior  expert. 

The  Break-Out  Groups  discussed  a  number  of  general  domain  features  and  issues,  and  went  into 
some  depth  in  discussing  certain  domains.  A  variety  of  domains  were  mentioned  and  discussed, 
including.  Software  engineering,  STEM  fields  (Science,  Technology,  Engineering,  and 
Mathematics),  Cybersecurity,  Cultural  Awareness,  Personnel  management,  UAV  operation. 
Weather  forecasting.  General  command. 

Many  emerging  military  domains  involve  coping  with  dynamism.  Many  involve  the  fusion  of 
information  is  a  characteristic  of  some  domains.  An  example  is  cyber  security.  People  must  take 
a  bunch  of  ambiguous  information  about  dynamically  evolving  situations  and  structure  it  and  put 
it  together  to  generate  an  outcome.  Many  new  military  domains  involve  training  competencies 
versus  training  tasks.  Eor  example,  Iraqi  cultural  training  caused  soldiers  to  think  more  and  draw 
from  experience  when  actually  deployed. 

Strategic  considerations  in  identifying  domains  for  study  would  include  selecting: 

•  A  tedious  job  does  not  make  for  high  level  of  motivation,  and  this  would  be  a  factor 
working  against  the  achievement  of  high  levels  of  proficiency. 

•  Domains  that  relate  to  STEM  fields  would  make  an  accelerated  learning  project 
strategically  linked  to  a  broader  national  concern. 

•  Domains  that  can  be  imported  into  academic  laboratories  for  study, 

•  Domains  that  could  be  treated  as  "open  source"  for  getting  data  on  baseline  performance, 

•  Domains  in  which  there  are  important  tasks  that  can  be  modeled  using  simple  linear 
models  and  in  which  such  models  perform  as  well  as  experts.  Training  can  involve 
“beating  the  linear  model”  as  a  test  of  proficiency  or  expertise. 

Maintenance  is  a  potential  domain  that  fits  these  criteria.  We  know  from  studies  of  this  domain 
that  one  can  chart  the  development  of  proficiency.  Troubleshooting  a  particular  device  would  be 
like  forming  a  "mini-career."  This  may  not  be  a  representative  domain,  however,  because  more 
parts  are  now  disposable  and  also  because  the  practitioner  can  rely  on  rely  on  manuals  rather 
than  critical  thinking.  Debugging  computer  programs  might  be  a  more  representative  domain  in 
this  regard. 

Cybersecurity  has  prospects  and  is  a  current  need.  There  are  universities  that  have  programs,  and 
one  might  study  their  students.  The  domain  has  a  technical  aspect,  but  also  includes  pattern 
recognition  skill  and  causal  reasoning.  This  domain  certainly  has  the  dynamism:  it  changes 
quickly.  Retention  might  be  an  issue  just  due  to  the  changes  in  domain.  (This  is  an  empirical 
question.)  The  challenge  will  be  to  coming  up  with  sub-domains  with  sufficient  complexities  to 
monitor  performance.  In  terms  of  security  issues,  it  may  be  difficult  to  get  actual  data  on 
performance  (across  a  proficiency  spectrum).  There  is  a  desire  to  avoid  exposing  workforce  or 
capability  limitations,  or  expose  mistakes  or  vulnerabilities.  Also,  because  this  domain  is  so 
dynamic,  and  new  technological  capabilities  are  always  coming  on  line,  training  or  acceleration 
may  have  a  limited  viable  lifetime:  One  cannot  anticipate  what  someone  may  be  able  to  extract 
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from  that  data  just  few  years  in  the  future.  Trust  in  automation  may  be  a  researchable  area. 
Although  it  does  not  represent  an  identifiable  career  or  subdomain  of  specialization. 

Emergency  Response  is  a  good  candidate  domain.  One  might  set  up  critical  emergency  response 
teams,  composed  of  experienced  practitioners  or  of  trainees.  Performance  could  be  evaluated  of 
novices  and  of  trainees  who  receive  the  same  data  as  experts. 

Search  and  Rescue/Medical  Evacuation  is  a  good  candidate  domain.  It  is  tactical  and  there  is  a 
great  deal  of  technological  change. 

Electric  Power  Generation  and  Distribution  is  a  good  candidate  domain.  The  U.  S.  Army  Corps 
of  Engineers  (CoE)  provides  power  generation  and  distribution  services  in  deployments  at  all 
scales.  In  addition,  CoE  engineers  provide  services  and  support  to  regional  utilities  in  areas  of 
operations.  Given  this  military  analog  to  the  civilian  utilities,  and  given  the  emerging  importance 
of  proficiency  training  in  the  utilities,  this  might  be  an  appropriate  domain  for  studies  of 
accelerated  proficiency. 

Cultural  Awareness  and  Skill  is  a  good  candidate  domain.  Cultural  decision  making  is  difficult 
for  the  soldier,  who  is  trained  to  be  in  war  then  must  transition  to  negotiation  tasks.  One  could 
run  terrain  teams  (anthropologist,  sociologist,  etc.)  through  scenarios  to  work  out  cultural  issues, 
and  see  what  they  do.  The  military  needs  more  "cultural  chameleons"  for  collaboration  (social 
adaptivity  and  reasoning)  sand  also  for  counterinsurgency  and  stability  operations. 

UAV  Operation  is  a  good  candidate  domain.  E-16  drawdown  and  UAV  ramp-up  is  an 
opportunity  to  study  as  there  will  be  pressure  to  make  E-16  utilization  more  specialized  and  more 
efficient.  Because  UAVs  are  becoming  so  capable,  the  nature  of  conflict  is  changing,  as  the 
nature  of  intelligence  operations.  UAVs  will  be  entering  the  private  sector.  A  current  design 
issue  is  whether  work  methods  and  software  systems  can  permit  a  single  operator  to  "fly"  more 
than  one  UAV,  so  the  nature  of  the  UAV  operator  job  is  likely  to  remain  a  changing  one. 
Because  of  the  technology  of  UAV  is  changing,  it  is  just  like  a  box,  assuming  a  person  can  fly  4- 
5  of  different  ones.  There  is  significant  emphasis  on  the  achievement  of  proficiency.  The 
equipment  is  so  expensive,  without  skill  training,  it  will  be  a  disaster. 

Weather  Eorecasting  is  a  good  candidate  domain.  It  may  be  appropriate  to  consider  the  area  of 
weather  forecasting  and  analysis  as  one  of  the  subject-matter  areas  to  research.  Several  factors 
recommend  this,  including  its  importance  to  all  of  the  military,  its  complexity,  its  inherently 
team-based  nature,  and  criticality  of  advanced  pattern  perception  skills.  There  is  an  ongoing  test 
bed  of  instructional  design  and  development  that  exists  in  the  COMET  Project  of  the  University 
Corporation  for  Atmospheric  Research  at  Boulder,  Colorado.  This  project  has  developed  a 
massive  body  of  training  material,  which  is  used  by  a  very  large  international  audience.  It 
represents  a  large  and  growing  design/development  activity  that  likes  to  partner  with  special 
needs.  Among  the  project’s  sponsors  already  are  the  Army,  Navy,  Air  Eorce,  and  Marines.  The 
COMET  project  would  provide  an  excellent  collaborative  partner,  able  to  supply  a  very  large 
number  of  participants  who  represent  an  appropriate  target  population  and  a  ready-made 
distribution  system  for  trying  out  and  iterating  research  products.  In  addition,  there  are  a  number 
of  excellent  undergraduate  and  graduate  education  programs  and  these  are  typically  interested  in 
collaboration  related  to  workforce  issues. 
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Table  4.1  expresses  the  Working  Meeting  consensus  about  criteria  for  domain  selection.  One 
especially  noteworthy  consideration  is  with  regard  to  knowledge,  skill,  and  reasoning.  Most 
academic/laboratory  studies  of  learning  involve  declarative  memory  (knowledge)  and  reasoning 
rather  than  motor  or  procedural  skills.  Domains  (tasks  or  jobs)  that  involve  a  mix  of  the 
knowledge,  skill,  and  reasoning  are  relevant  for  many  jobs,  but  especially  so  for  military  jobs 
(piloting,  marksmanship,  mine  detection,  and  so  forth). 

Table  4.1.  Conclusions  regarding  criteria  for  selecting  domains. 


The  task/job  work  requires  highly  organized  knowledge,  reasoning  skills,  and  motor  or 

procedural  skills  as  well. _ 

The  job/task  work  involves  coping  with  dynamics  and  complexity. _ 

Achievement  of  proficiency  involves  practice  at  tough  cases. _ 

The  work  depends  on  skill  at  pattern  learning  (recognition-primed  decision  making), 

sensemaking,  and  the  other  primary  macrocognitive  functions. _ 

Practitioner  motivation  and  affect  play  a  key  role. _ 

Domains  requiring  a  capacity  to  be  adaptive  and  resilient. _ 

Domains  with  recently  transformed  strategic  environment,  where  people  have  more  than  one 

duty  assignment. _ 

The  work  involves  teaming  of  specialists,  and  requires  expert  management  of  teams. _ 

There  is  organizational  support  for  processes  of  achieving  higher  levels  of  proficiency. _ 

Domains  match  into  STEM  (science,  technology,  engineering  and  mathematics). _ 

Ease  of  study  (factors  including  data  access,  availability  of  known  and  valid  measures,  etc.). 


4.2  Paradoxes  of  Training  and  Paradoxes  of  Careers 

Looking  across  careers  and  domains  that  have  been  studied,  people  who  are  highly  motivated 
become  experts  and  those  who  are  not  intrinsically  motivated  are  less  likely  to  become  experts. 
Individuals  who  feel  a  personal  investment  are  more  likely  to  look  at  other  various  aspects  of 
their  domain  that  are  not  necessarily  central  to  their  specific  job  or  roles.  They  actively  seek  an 
ever- widening  scope  for  their  expertise. 

Organizational  customs  and  policies  often  cut  against  this.  For  example  in  the  intelligence 
domain,  one  either  stays  in  and  moves  up,  or  gets  out.  The  motivation  shifts  from  getting  better 
at  analytical  roles  to  getting  better  at  administering.  And  yet  ironically,  the  individuals  who  are 
motivated  tend  to  be  the  ones  who  volunteer  to  get  extra  training.  Is  there  support  in  the 
organization  to  pursue  expertise,  an  organizational  culture  that  fosters  workers  to  become  experts 
in  something  related  to  the  overall  organizational  mission? 

There  are  a  number  of  paradoxes  and  ironies  of  learning,  training  and  careers  that  complicate  any 
effort  toward  acceleration  of  proficiency  and  the  facilitation  of  retention.  We  call  out  these 
ironies  and  paradoxes  by  beginning  with  two  career  case  studies  that  were  discussed  at  the 
Working  meeting. 
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A  Career  Case  Study:  Military  Intelligence 

By  the  time  of  achieving  senior  rank,  an  intelligence  officer  will  have  spent  as  much  as  half  their 
time  engaged  in  work  other  than  intelligence  analysis.  This  would  be  an  impediment  to  the 
achievement  of  high  levels  of  proficiency.  Of  the  time  spent  in  analytical  jobs/roles,  a  majority 
of  the  assignments  would  be  in  one  type  of  intelligence  analysis  (Signals,  Imagery,  Targeting, 
UAV,  etc.).  Targeteers  and  Weapons  Officers  do  stay  in  that  assignment  for  long  periods  and  do 
become  experts.  But  most  jobs  do  not  involve  consecutive  assignments. 

Time  spent  in  non-analytical  assignments  involve  work  at  a  higher  level  (e.g.,  global  ISR 
operations,  planning  and  assessment).  Each  of  the  high-level  roles/jobs  has  multiple  core 
competency  areas.  It  takes  time  to  achieve  competency,  and  there  is  never  enough  time  for  study. 
You  cannot  have  consecutive  assignments  in  the  same  core  competence. 

Then  after  some  years  of  work  at  the  higher-level,  job  one  gets  transferred  back  to  some 
particular  analytical  role.  Due  to  the  shortage  for  field-grade  officers,  there  is  a  tendency  to  seek 
and  be  assigned  roles  that  fall  within  prior  expertise.  This  would  contribute  to  the  progress 
toward  higher  levels  of  proficiency  at  that  analytical  role.  “But  when  you  get  back  [to  the 
previous  analytical  role]  no  one  knows  you.  You  have  to  rebuild  your  reputation.  Mentors  are  no 
longer  co-located  with  you.  ” 

A  Career  Case  Study:  Aircrew  Expertise 

Twelve  years  of  operational  assignments  is  the  country’s  return  on  investment  for  all  the 
training.  Any  time  after  two  years  at  the  principal  assignment,  an  individual  can  be  moved  off 
that  career  path  to  do  something  else,  typically  something  that  is  not  flying.  It  can  be  teaching, 
UAV  piloting,  or  a  staff  job.  Eor  some  individuals  experiencing  a  longer  hiatus,  they  are  brought 
back  to  wingman  status  and  in  the  training,  are  basically  starting  the  career  path  all  over  again. 

In  an  example  career,  an  officer  was  a  pilot  on  B-52s  and  then  transitioned  as  navigator  to  the  B- 
1.  The  transition  course  has  three  tracks,  based  on  prior  experience,  that  is,  flight  hours. 

In  this  example  career,  each  assignment  involved  over  1,000  hours  flying  time.  Including  the 
planning  and  after-action  activities,  total  time  approximates  to  5,000  hours  of  at-the-job 
experience.  Aircrew  members  achieved  proficiency  after  their  first  assignment,  and  higher  levels 
of  proficiency  after  their  second  assignment  (or  after  weapons  school).  This  contrasts  with  the 
current  situation.  Eewer  flying  hours  are  available  (due  to  expense),  so  there  is  the  pressure  of 
getting  experience  with  less  practice  time.  And  there  is  the  staffing  pressure.  In  this  case,  since 
there  is  significant  functional  overlap  of  the  B-52  and  the  B-1,  transfer  is  rapid.  Thus,  the 
individual  in  this  example  career  was  told  he  would  be  an  instructor  immediately  after  taking  his 
B-1  qualifying  examination. 

In  the  example  career,  there  was  a  hiatus  during  the  B-1  assignment.  That  hiatus  required  the 
requalification,  including  15  practice  flights. 

"Then  after  two  sorties  I  felt  back  up  to  speed.  I  was  at  least  as 
proficient  as  the  others....  [but]  stuff  that  had  changed  in  the  B-1 
was  what  took  longer/most  time  to  (re)acquire.  For  software 
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especially  for  new  weapons  1  will  have  an  instructor  brief  me  and 
take  me  out  on  my  first  sortie,  tell  me  the  buttonology.  ” 

This  comment  has  clear  implications  for  the  content  of  training  given  during  hiatus  periods. 

Based  on  these  and  other  career  case  studies  discussed  at  the  Working  meeting,  we  can  list  some 
ironies  and  paradoxes  of  learning,  training  and  careers  that  complicate  any  effort  toward 
acceleration  of  proficiency  and  the  facilitation  of  retention.  These  are  described  in  Tables  4.2  and 

4.3. 
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Table  4.2.  Paradoxes  and  ironies  of  learning  that  complicate  the  acceleration  effort. 


Paradox  of  Domain 
Specificity 


The  Envisioned 
World  Paradox 


The  Moving  Target 
Paradox 


Paradox  of 
Feedback 


The  Skills- 
Capacities  Paradox 


Paradox  of  Tough 
Tasks 


The  Paradox  of 
Failure 


The  Irony  of 
Evaluation 


High  proficiency  is  domain  and  sub-domain  specific.  Generalization  about 
transfer  of  either  knowledge  or  skill  is  always  tentative.  On  the  other 
hand,  expertise  is  defined  by  experience  at  coping  with  diverse  situations 
that  stretch  the  competence  envelope,  so  there  must  be  transfer. 

Changing  world  circumstances  entail  entirely  new  forms  of  work.  The 
need  for  integrated  operations  (airs,  space,  and  cyber)  is  apparent,  and  the 
need  for  a  workforce  at  the  highest  levels  of  proficiency  is  urgent.  But 
cross-domain  integrated  training  is  not  realistic.  There  is  too  much  to 
train,  in  too  little  time.  The  pace  of  adaption  and  updating  of  instruction 
cannot  keep  pace  with  the  change  in  the  work.  One  cannot  plan  in  advance 
for  training  needs  or  for  retraining  needs. 

Also,  as  technology  changes,  as  assignments  change,  and  as  jobs  change, 
the  cognitive  work  changes.  Fearning  and  re-learning  on  the  job  must  be 
continuous.  But  there  is  too  much  to  train,  in  too  little  time.  Thus,  the 
pressure  is  to  do  “just-in-time  training,”  but  such  training  is  at  odds  with 
notions  of  proficiency  training. 

You  need  to  provide  rich,  informative  feedback  about  both  outcome  and 
process,  but  feedback  can  forestall  the  learner’s  own  drive  to  achieve 
understanding  by  thinking  about  past  performance.  Some  kinds  of 
immediate  feedback  help,  some  don’t  some  kinds  of  delayed  feedback 
help,  some  don’t. 

Analysis  of  jobs  by  skill  (task  analysis)  generates  process  descriptions  that 
can  be  used  in  training  and  can  be  used  as  prescriptions  or  roadmaps  for 
work  conducted  in  the  operational  context,  but  will  be  brittle  due  to 
“moving  target”  phenomena.  Analysis  of  jobs  by  capacity  (e.g., 
sensemaking,  coordinating,  planning,  etc.)  is  more  appropriate  for  the 
analysis  of  complex  cognitive  work  systems  but  feeds  into  training  only 
on  the  assumption  that  instruction  in  “critical  thinking”  will  transfer. 
Achievement  of  high  levels  of  proficiency  stems  from  extensive  and 
motivated  practice  at  difficult  or  tough  cases.  But  case  difficulty  is  closely 
tied  to  novelty  or  “lying  at  the  fringes  of  the  ordinary,”  and  thus  the 
learning  would  be  to  some  extent  case  specific.  Thus,  it  might  not  promote 
transfer  though  the  “tough  tasks”  assumption  implies  that  it  does. 

Different  things  are  learned  in  different  ways  by  individuals  at  different 
levels  of  proficiency.  If  a  trainee  fails  or  errs  when  working  on  a  scenario, 
the  learning  will  likely  be  about  how  to  avoid  that  particular  error  and  will 
be  to  some  extent  unique  to  the  exercise.  But  journeymen,  senior 
journeymen,  and  experts  learn  more  from  their  mistakes  than  their  correct 
actions/decisions. 

Evaluation  should  be  a  learning  experience,  and  yet  this  aspect  of 
performance  or  knowledge  evaluation  is  generally  not  leveraged  as  useful 
feedback. 
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Another  problem  that  was  noted  in  the  Working  Meeting  was  the  ambiguity  of  some  core 
concepts  and  the  linkage  of  these  to  deeper  philosophical  and  methodological  issues.  Training, 
transfer,  proficiency  must  all  be  thought  of  in  terms  of  types  of  knowledge  (declarative, 
procedural,  tacit,  intuitive,  etc.).  In  some  circumstances  these  distinctions  are  useful  (e.g.,  the 
generalization  that  well-learned  skills  decay  less  than  declarative  knowledge)  but  in  other 
circumstances  they  are  not  (e.g.,  the  problem  of  operationally  defining  intuition). 

An  example  discussed  at  the  Working  Meeting  was  the  concept  of  a  “task.”  Research  by  Alice 
Healy  and  her  colleagues  (see  Healy,  2007)  had  college  student  participants  learn  a  time 
estimation  task  (estimating  the  durations  of  temporal  intervals)  and  simultaneously  attempt  to 
recite  the  alphabet  backwards.  After  learning  to  a  level  of  minimal  proficiency,  the  secondary 
task  was  changed.  Participants  resultantly  performed  worse  at  the  primary  task.  So,  when  they 
were  learning  the  dual  tasks  (the  experimenter’s  perspective)  were  they  not  just  learning  a  single 
“task”  (the  participant’s  perspective)?  Now  consider  the  notion  of  task  with  regard  to  complex 
sociotechnical  systems.  Cognitive  work  is  the  context-sensitive,  knowledge-driven  choice  among 
alternative  activities.  It  often  appears  as  if  workers  are  conducting  tasks,  but  that  is  in  routine 
situations  for  stable  task  goals.  A  “task,”  in  the  traditional  human  factors  sense,  is  a  sequence  of 
specifiable  activities  that  will  achieve  some  particular  goal.  Such  task  analyses  can  be  used 
prescriptively,  and  can  be  used  in  training.  But  it  is  not  judicious  to  take  this  as  the  single  model 
for  founding  a  theory  of  instruction  for  complex  cognitive  work. 

A  specific  example  of  how  the  traditional  concept  of  “task”  may  be  an  impediment  to  advances 
in  research  and  theory  is  that  it  reinforces  an  artificial  notion  of  separability.  Complex  cognitive 
work  can  always  be  analyzed  into  component  goals  and  those  goals  mapped  onto  activities  and 
sometimes  onto  step-wise  procedures.  But  those  activities  are  more  often  parallel  and  integral  (a 
single  activity  contributes  to  the  achievement  of  more  than  one  goal)  and  are  sometimes  in 
conflict  even  when  they  are  integral  (a  single  activity  contributes  to  one  goal  but  at  the  expense 
of  another). 

An  example  might  help.  A  weather  forecaster  is  examining  upper  air  charts  radar  to  look  for 
signs  of  instability.  This  is  a  “primary  task”  in  that  it  has  a  specifiable  goal  and  describable 
activities  associated  with  achieving  it.  While  looking  at  the  upper  air  charts,  the  forecaster  can 
also  examine  a  Doppler  radar  image  to  look  for  indicators  of  whether  advection  is  occurring. 
This  too  is  a  “task”  in  that  it  has  a  specifiable  goal  and  describable  activities  associated  with 
achieving  it.  While  examining  the  radar  the  forecaster  is  sensemaking  with  reference  to  his 
mental  model  based  on  inspection  of  the  upper  air  data.  For  the  forecaster,  the  real  task  is  to 
understand  the  atmospheric  dynamics  and  project  them  into  the  future.  It  is  a  distortion  to 
express  in  the  psychologist’s  language  of  tasks  (“Look  here  to  find  this  cue,  look  there  to  find 
that  cue,  etc.”). 

The  traditional  notion  of  task  reinforces  the  view  that  in  some  complex  activity  there  is  a 
“primary”  task  and  one  or  more  “secondary  tasks.”  Such  analysis  makes  for  comfortable 
experimental  design  in  the  academic  laboratory.  It  makes  for  a  comfortable  decomposition  into 
sequences.  But  it  is  of  limited  applicability  to  complex  cognitive  work.  Which  task  counts  as  the 
primary  one  and  which  as  the  secondary  ones  varies  over  time  and  depends  on  circumstance  and 
the  unfolding  of  events. 
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Therefore,  one  should  question  the  wording  of  the  functional  task  principle  when  applied  to 
accelerating  proficiency.  By  the  principle,  secondary  task  requirements  are  often  integrated  with 
primary  task  requirements  during  learning,  resulting  in  the  acquisition  of  a  single  functional  task 
rather  than  two  separate  tasks  (Healy,  et  al.,  2005;  Hsiao  &  Reber,  2001).  As  the  forecasting 
example  suggests,  primary  and  secondary  tasks  are  always  integrated,  so  much  so  that  the 
primary-secondary  distinction  loses  its  value.  The  concept  of  the  “extraneous”  task  definitely 
loses  its  value.  Certainly,  if  some  job  involves  some  principle  task  ("primary”  goals)  and 
additional  tasks  not  deemed  of  surmount  importance  (“secondary”  goals),  it  nonetheless  is 
necessary  to  train  in  all  the  tasks  and  do  so  in  a  way  that  realistically  integrates  rather  than 
separates  them.  In  short:  If  in  the  actual  job  the  worker  has  to  do  two  things  at  once,  then  training 
should  involve  doing  two  things  at  once.  What  counts  as  a  “task”  depends  entirely  on  the 
perspective  (experimenter/job  analyst  versus  the  worker). 
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Table  43.  Paradoxes  and  ironies  of  careers  that  complicate  the  development  of  proficiency. 


The  Policy- 
Proficiency 
Paradox 


The  Culture- 

Measurement 

Clash 


The  Myth  of  the 
Career  Path 


The  Irony  of 
Transitioning 


Table  continues 


In  some  domains  (e.g.,  intelligence  analysis,  cultural  understanding,  others) 
broad  knowledge  is  a  requirement.  Promotion  to  leadership  positions 
depends  on  having  had  experience  in  diverse  jobs/roles.  But  this  policy 
limits  the  opportunity  for  individuals  to  achieve  high  levels  of  proficiency 
in  any  one  job/role. 

Fighter  piloting  entails  the  issue  of  measuring  performance  gains  after  the 
achievement  of  ceiling-level  performance.  Research  has  shown  that  experts 
continue  to  get  better  even  after  they  “always  get  it  right”  on  some  criteria 
measure  of  performance.  Once  a  pilot  has  achieved  very  good  performance 
as  evaluated  at  some  stage  of  training  or  operational  performance,  they  will 
maintain  that  level,  on  the  record,  because  if  they  do  not  they  will  not 
advance  in  their  military  career.  This  suggests  that  in  some  domains  or 
organizations  there  might  actually  be  incentives  against  measuring  (and 
thereby  understanding)  proficiency  and  its  development. 

Careers  are  not  pathed  in  the  sense  of  tracking  a  standardized  progression 
plan.  “While  there  is  a  written  path,  it  is  rarely  followed.  ”  Careers  are 
managed  within  sets  of  changing  constraints,  especially  “bellybutton 
shortages.”  In  some  specialties  (e.g.,  piloting)  there  is  a  sophisticated 
process  for  managing  careers.  In  most  specialties  there  is  not.  “You  can 
select  your  preferences  for  future  assignments,  but  your  commander  also 
makes  recommendations  and  then  the  personnel  office  decides,  depending 
on  needs,  billets  available,  etc.  ”  In  some  specialties,  time  out  of  career 
field  is  not  tracked  so  you  do  not  know  when  to  add  reintegration  training. 

Promotion  hinges  on  broad  experience,  to  develop  competency  at 
jobs/roles  that  fall  at  supervisory  levels.  But  assignments  (in  the  military) 
will  transition  individuals  across  levels.  Just  in  time  training  courses  are 
out  there,  but  are  not  designed  for  people  transitioning  across  levels. 
Training  is  necessary  prior  to  any  reassignment,  and  while  the  courses 
(typically  2  months  long)  can  be  good  for  exposure  they  do  not  promote 
deep  comprehension.  There  is  not  enough  time  available.  There  is  no 
tracking  of  people  who  can  help  in  training.  It  can  be  awkward  to  ask  for 
help  form  subordinates. 
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Table  continued 


The  Ironies  of 
Demotivation 


The  Ironies  of 
Reassignment 


The  Ironies  of 
Under-utilization  of 
Resources 


The  Irony  of  the 
Consequences  of 
Limiting  the 
Resources 


Demotivation  manifests  in  a  number  of  forms  and  for  a  variety  of  reasons. 
Deployment  itself  contributes  to  the  acceleration  of  proficiency,  but  the 
basic  job  responsibilities  can  consume  one’s  time  and  energy.  There  is 
significant  burn  out  potential  by  doing  extra  learning  and  training. 
Broadening  negatively  impacts  career  progression  and  triggers  thoughts  of 
getting  out.  “Tom  repeatedly  experience  a  loss  of  confidence.  ” 

Reassigning  an  individual  just  as  they  achieve  high  proficiency  cuts 
against  the  notion  that  organizational  capability  is  built  upon  individual 
expertise.  Reassignment  may  be  necessary  for  staffing,  and  is  necessary 
for  training  individuals  who  are  selected  to  move  to  higher  levels  of  work. 
But  it  can  be  recognized  that  reassignment  is  at  least  partly  responsible  for 
creating  the  problem  of  skill  decay  in  the  first  place.  There  are  significant 
secondary  effects  that  also  should  be  considered.  For  example,  upon 
return  to  a  primary  assignment  the  post-hiatus  refresher  training  might 
involve  work  on  routine  cases.  This  may  lead  to  overconfidence  in  the 
earliest  stage  of  a  redeployment,  and  might  limit  the  worker’s  ability  to 
cope  with  novel  or  challenging  situations. 

While  some  organizations  recognize  the  need  to  collect  and  share  “lessons 
learned,”  such  lessons  are  often  archived  in  ways  that  make  meaningful 
search  difficult,  and  hence  become  “lessons  forgotten”  Debriefing  records 
are  a  significantly  under-utilized  resource  for  training,  and  could  be 
formative  of  a  library  of  cases,  lessons  learned,  and  scenarios  for  use  in 
training. 

In  past  years,  re-training  following  hiatus  could  be  brief  (6  weeks)  (we 
might  say  “accelerated”)  because  pilots  had  had  thousands  of  hours  of 
flying  time  prior  to  their  hiatus.  But  pilots  are  flying  less  now,  largely  due 
to  high  cost.  This  has  two  repercussions:  It  mandates  more  training  post 
hiatus  (which  has  its  own  associated  costs)  and  the  reduced  flight  practice 
hours  detract  from  the  goal  of  accelerating  the  achievement  of  proficiency, 
thus  placing  the  overall  organizational  capability  at  risk. 


4.3  Research  Challenges 


Much  of  the  Working  Meeting  discussion  can  be  integrated  under  this  theme,  with  particular 
topics  being: 

o  Forging  the  Right  Theory 

o  Finding  the  Right  Training  and  Mentoring  Methods 
o  Finding  Methods  and  Designs  for  Acceleration  Research 

4.3. 1  Forging  the  Right  Theory 

Given  the  complexity  of  many  domains,  accelerating  the  development  of  proficiency  cuts  across 
processes,  ranging  from  perception  and  memory  to  category  learning,  problem  solving,  and 
decision  making.  As  such,  any  viable  research  program  needs  to  put  forth  a  theoretical  approach 
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that  can  encompass  a  wide  range  of  cognition  (see  Roediger,  Gallo,  &  Geraci,  2002).  There 
appears  to  be  some  significant  degree  of  consensus  concerning  the  important  theoretical  notions 
to  scaffold  our  understanding  of  the  achievement  of  proficiency.  Before  expressing  that 
consensus  we  must  point  out  a  topic  of  discussion  at  the  Working  Meeting  on  which  there  was 
not  a  consensus. 

Cognitive  Load  Theory  asserts  that  there  are  different  kinds  of  mental  workload  (Cooper,  1998; 
Paas,  Renkl,  &  Sweller,  2003;  Sweller,  van  Merrienboer,  &  Paas,  1998;  van  Merrienboer  & 
Sweller,  2005).  Intrinsic  cognitive  load  depends  on  the  complexity  and  interactivity  of  the 
materials  or  tasks  that  must  be  learned.  Intrinsic  cognitive  load  is  the  difficulty  associated  with 
the  problem  type  and  cannot  be  altered  by  an  instructor.  Extraneous  cognitive  load  is  caused  by 
activities  during  learning  that  are  unnecessary  and  so  interfere  with  schema  acquisition  and 
automatization.  Extraneous  cognitive  load  is  created  by  the  manner  in  which  information  is 
presented  to  learners,  attributed  to  the  design  of  the  instructional  materials.  This  is  under  the 
control  of  instructors  and  instructional  designers.  Germane  cognitive  load  is  workload  to  achieve 
the  goals  of  the  task.  Germane  activities  are  beneficial  for  the  advancement  of  proficiency. 
Germane  cognitive  is  that  load  devoted  to  the  processing,  construction  and  “automatization”  of 
schemata — a  mixture  of  phenomena  which  we  might  interpret  as  the  effort  required  to  form  a 
mental  model.  How  this  differs  substantively  from  intrinsic  cognitive  load  is  a  matter  of  debate. 

The  way  this  translates  from  the  academic  context  to  the  more  practical  context  of  proficiency 
acceleration  is  not  clear,  in  part  because  the  distinctions  are  not  entirely  clear.  The  purported 
value  of  these  distinctions  is  guidance  in  the  analysis  of  cognitive  work.  The  literal  claim  is  that 
intrinsic  load  is  immutable  but  instructional  designers  can  manipulate  extraneous  and  germane 
load.  It  is  suggested  that  they  limit  extraneous  load  and  promote  germane  load.  In  other  words, 
instructional  processes  should  minimize  extraneous  cognitive  load  and  optimize  germane 
cognitive  load.  That  might  seem  like  a  homily  with  which  no  one  could  disagree,  but  is  it 
judicious  in  practice?  Actual  work  in  complex  cognitive  work  systems  is  typified  by  distractions, 
the  need  for  kluges,  and  the  need  for  work-arounds  (Koopman  &  Hoffman,  2003).  If  training 
does  not  prepare  workers  for  that  “extrinsic”  mental  workload,  and  focuses  solely  on  preparation 
to  engage  in  the  activities  that  contribute  directly  to  the  achievement  of  the  primary  task  goals, 
then  the  training  will  not  adequately  prepare  workers  for  “the  real  world.”  In  other  words,  there 
will  be  a  boundary  condition  imposed  on  transfer. 

Turning  now  to  the  consensus,  it  is  agreed  that  Recognition-Primed  Decision  making  is  a 
phenomenon  that  typifies  expertise,  in  diverse  domains.  It  is  agreed  that  a  core  phenomenon  of 
proficiency  is  the  ability  to  apprehend  complex  dynamic  patterns. 

All  of  the  pertinent  theories  are  essentially  sets  of  assertions  or  postulates.  Erom  the  standpoint  of 
formal  philosophy  of  science,  they  often  lack  metatheoretical  and  ontological  assertions  and 
assertions  about  methodology.  Hence,  they  should  be  thought  of  not  as  theories,  but  as  sets  of 
hypotheses  and  postulates  about  concepts,  and  one  can  be  at  liberty  to  combine  or  merge  their 
proposition  sets,  as  we  do  in  the  following  discussion. 

It  is  also  agreed  that  the  theoretical  foundations  for  accelerated  proficiency  will  include  notions 
from  Cognitive  Elexibility  Theory  (GET)  (Spiro,  1988,  1992)  and  Cognitive  Transformation 
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Theory  (CTT)  (Klein  &  Baxter,  2009).  Each  of  these  theories  presents  as  a  set  of  hypotheses,  and 
these  can  be  readily  integrated.  The  key  ideas  of  each  of  these  theories  are  presented  in  Tables 
4.4  and  4.5. 


135 


Report  on  FA8650-09-2-6033 


Table  4.4.  Key  ideas  of  Cognitive  Flexibility  Theory. 


Core  Syllogism 

1.  Learning  is  the  active  construction  of  knowledge. 

2.  Learning  entails  reductive  explanation. 

3.  Reductive  explanation  reinforces  and  preserves  itself  through  misconception  networks  and 
through  knowledge  shields. 

4.  Advanced  learning  is  the  ability  to  flexibly  apply  knowledge. 

Therefore,  instruction  by  incremental  complexification  will  not  be  conducive  of  advanced 
learning. 

Therefore,  advanced  learning  is  promoted  by  emphasizing  multiple  interconnectedness  of  cases 
and  concepts  along  multiple  conceptual  dimensions,  and  the  use  of  multiple  representations. 

Empirical  Ground  and  Claims 

Studies  of  learning  of  topics  having  conceptual  complexity  (medical  students). 

Demonstrations  of  knowledge  shields  and  the  dimensions  of  difficulty. 

Demonstrations  that  learners  tend  to  oversimplify  (reductive  bias)  by  the  spurious  reduction  of 
complexity. 

Studies  of  the  value  of  using  multiple  analogies. 

Learners  tend  to  rely  too  much  on  generic  abstractions,  which  leads  to  failure  to  transfer 
knowledge  to  new  cases. 

Learners  tend  to  regularize  that  which  is  irregular,  which  leads  to  failure  to  transfer  knowledge  to 
new  cases. 

Learners  tend  to  de-contextualize  concepts,  which  leads  to  failure  to  transfer  knowledge  to  new 
cases. 

Learners  take  the  role  of  passive  recipient  versus  active  participants. 

Conceptual  complexity  and  case-to-case  irregularity  pose  problems  for  traditional  theories  and 
modes  of  instruction. 

Instruction  that  simplifies  (the  incremental  complexification  approach)  can  detract  from 
advanced  knowledge  acquisition  by  facilitating  the  formation  of  reductive  understanding  and 
knowledge  shields. 

Instruction  that  emphasizes  recall  memory  will  not  contribute  to  inferential  understanding  and 
advanced  knowledge  acquisition  (transfer). 

Additional  Propositions  in  the  Theory 

Advanced  knowledge  acquisition  (apprentice-journey man-expert)  depends  on  ability  to  achieve 
deeper  understanding  and  apply  it  flexibly. 

Barriers  to  advanced  learning  include  complexity,  interactions,  context-dependency,  and  ill¬ 
structuredness  (inconsistent  patterns  of  concepts-in-combination). 

Active  "assembly  of  knowledge"  from  different  conceptual  and  case  sources  is  more  important  in 
learning  (for  domains  of  complexity  and  ill-structuredness)  that  retrieval  of  knowledge 
structures. 

Table  continues 
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Table  continued 


Misconceptions  compound  into  networks  of  misconceptions.  Misconcepts  of  fundamental 

concepts  can  cohere  in  systematic  ways,  making  each  misconcept  easier  to  believe. _ 

Representations  with  high  interconnectedness  will  tend  to  serve  as  "misconception-disabling 

correct  knowledge." _ 

Cognitive  flexibility  is  the  ability  to  mobilize  potential  knowledge,  small,  pre-compiled 

knowledge  structures  ("adaptive  schema  assembly"). _ 

Cognitive  flexibility  is  the  ability  to  represent  knowledge  from  different  conceptual  and  case 
perspectives  and  construct  from  those  a  knowledge  ensemble  tailored  to  the  needs  of  the  problem 
at  hand. 


The  emphasis  of  CFT  on  overcoming  simplifying  mental  models  advises  against  applying 
instructional  methods  that  involve  progressive  compexification.  As  is  true  for  all  methods, 
whether  incremental  complexificatiuon  works  or  not  depends  upon  how  it  is  conducted 
(Reigeluth,  personal  communication).  A  traditional  learning  hierarchy  approach  that 
compartmentalizes  material  and  assumes  fixed  tasks  (e.g.,  Gagne,  1968)  can  detract  from 
advanced  knowledge  acquisition  because  the  decomposition  limits  opportunities  to  learn  about 
interactions.  The  Simplifying  Conditions  Method  (Kim  &  Reigeluth,  1996;  Reigeluth,  1999) 
provides  an  approach  to  incremental  complexification  that  is  ideally  suited  to  advanced 
knowledge  acquisition  because  it  incrementally  introduces  interactions  as  well  as 
"compartments"  of  to-be-learned  material. 
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Table  4.5.  Key  ideas  of  Cognitive  Transformation  Theory. 


Core  Syllogism 

1.  Learning  consists  of  the  elaboration  and  replacement  of  mental  models. 

2.  Mental  models  are  limited  and  include  knowledge  shields. 

3.  Knowledge  shields  lead  to  wrong  diagnoses  and  enable  the  discounting  of  evidence. 

Therefore  learning  must  also  involve  unlearning. 

Empirical  Ground  and  Claims 

Science  as  sensemaking. 

Flawed  "storehouse"  memory  metaphor  and  the  teaching  philosophy  it  entailed  (memorization  of 
facts;  practice  plus  immediate  feedback,  outcome  feedback.) 

Studies  of  science  learning  show  how  misconceptions  lead  to  error. 

Studies  of  scientist  and  student  reactions  to  anomalous  data. 

Success  of  "cognitive  conflict"  methods  at  producing  conceptual  change. 

Additional  Propositions  in  the  Theory 

Mental  models  are  reductive  and  therefore  incomplete  and  flawed. 

Learning  is  the  refinement  of  mental  models. 

Experts  have  more  detailed  and  more  sophisticated  mental  models  than  novices.  Experts  have 
more  accurate  causal  mental  models. 

Elawed  mental  models  are  barriers  to  learning  (knowledge  shields). 

Learning  is  by  sensemaking  (discovery,  reflection)  as  well  as  by  teaching. 

Refinement  of  mental  models  entails  at  least  some  un-leaming  (accommodation;  restructuring, 
changes  to  core  concepts). 

Refinement  of  mental  models  can  take  the  form  of  increased  sophistication  of  a  flawed  model, 
making  it  easier  for  the  learner  to  explain  away  inconsistencies  or  anomalous  data. 

Learning  is  discontinuous.  (Advancing  when  flawed  mental  models  are  replaced,  stable  when  a 
model  is  refined  and  gets  harder  to  disconfirm.) 

People  have  a  variety  of  fragmentary  mental  models.  "Central"  mental  models  are  causal  stories. 

The  merger  of  CFT  and  CTT  would  be  a  key  part  of  the  theoretical  foundations  for  acceleration 
research.  Problems  that  involve  dynamics,  interacting  processes,  nonlinear  causation, 
simultaneous  events,  and  other  features  are  particularly  difficult  for  learners.  When  confronted 
with  the  gaps  and  inaccuracies  in  their  knowledge  or  reasoning,  the  learners  are  likely  to  invoke 
a  “knowledge  shield”  that  allows  them  to  preserve  their  simplistic  understanding  in  the  face  of 
contradictory  evidence  (Feltovich,  et  al.,  2001).  This  links  to  the  core  concepts  of  CTT  (Klein  & 
Baxter,  2006),  which  developed  from  research  in  the  field  of  Expertise  Studies.  CTT  argues  for 
the  importance  of  “unlearning” — for  experiences  that  force  people  to  lose  faith  in  their 
entrenched,  simplistic  mental  models  so  that  they  can  move  to  deeper  levels  of  understanding. 

CFT  suggests  that  for  expertise  acceleration,  feedback  must  help  the  learner  transcend  their 
inclination  to  invoke  a  knowledge  shield,  and  CTT  suggests  that  feedback  must  help  the  learner 
un-learn  concepts  or  notions  that  incorrectly  simplify  their  understanding  of  the  domain.  It  is 
widely  believed  that  people  learn  more  from  their  mistakes  than  from  what  they  get  right.  When 
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everything  works  the  way  it  is  supposed  to,  one  is  less  likely  to  receive  feedback  about  what  did 
not  work  or  what  might  have  been  done  better.  Experts  seek  out  corrective  feedback,  feedback 
pointing  out  targets  for  improvement.  It  has  been  said  that  apprentices  make  the  same  mistake 
twice,  journeyman  make  the  same  mistake  once  and  experts  work  until  they  never  make 
mistakes.  While  this  is  a  point  well  taken,  domain  specialists  who  are  intrinsically  motivated 
often  seek  out  corrective  feedback  that  allows  them  to  perceive  their  errors.  Sonnentag  (2000) 
showed  that  the  more  experienced  problem  solvers  (domain  of  software  engineering)  sought  out 
corrective  feedback  from  coworkers. 

Cognitive  Transformation  Theory  emphasizes  the  paradoxical  fact  that  the  better  the  mental 
model,  the  harder  it  is  to  move  past  it  because  the  model  does  a  better  a  job  and  is  easier  to 
protect  using  knowledge  shields  to  explain  away  contrary  data.  Some  workers  stagnate  as  their 
performance  asymptotes  because  they  are  so  effective  in  defending  flawed  mental  models. 
Cognitive  Transformation  Theory  argues  for  the  importance  of  “unlearning”  -  experiences  that 
force  people  to  lose  faith  in  their  mental  models  so  that  they  can  move  to  the  next  level. 

The  implication  is  that  high  levels  of  proficiency  are  achieved  when  the  practitioner  has  an 
ability  to  lose  confidence  in  an  existing  mental  model. 

CFT  and  CTT  each  try  to  achieve  growth  but  in  different  ways.  For  CFT  it  is  flexibility  and  for 
CTT  it  is  a  better  mental  model,  but  one  that  will  have  to  be  thrown  out  later  on.  CFT  does  not 
say  what  the  sweet  spot  is  for  flexibility.  It  just  pushes  for  more  flexibility.  If  you  over 
complexify  you  do  not  get  any  traction  and  can  get  paralyzed.  It  thus  might  be  considered  a 
“lopsided”  theory,  or  at  least  an  incomplete  one.  Humans  are  very  flexible  but  need  structure  in 
order  to  gain  momentum.  To  use  the  analogy  of  the  movement  of  a  snake,  if  it  were  completely 
flexible  it  could  not  move. 

The  two  theories  have  slightly  different  focus.  The  two  theories  have  similar  core  syllogisms. 
Both  depend  on  the  notions  of  mental  model  formation  and  knowledge  shields.  They  both  tap 
into  the  same  general  empirical  base  about  the  phenomena  of  proficiency,  expertise,  and  high- 
end  learning.  Although  the  premises  of  CTT  seem  more  general  and  encompassing  than  those  in 
the  core  syllogism  of  CFT,  CFT  has  academic  and  scientific  precedent. 

Taken  separately,  they  are  incomplete  in  the  sense  that  they  tap  primarily  into  different  sets  of 
phenomena.  Thus,  if  merged  they  might  form  a  more  complete  theory.  The  two  theories  are 
certainly  compatible  and  we  find  no  outright  contradictions  in  comparing  their  propositions. 
Given  that  they  point  to  complementary  ideas  and  implications  about  training,  it  would  seem 
practical  to  merge  them. 

The  core  syllogism  of  the  merged  theory  would  be  the  following: 
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1.  Learning  is  the  active  construction  of  knowledge,  the  elaboration  and  replacement  of 
mental  models  or  causal  stories. 

2.  Learning  entails  reductive  explanation. 

3.  Mental  models  are  limited.  People  have  a  variety  of  fragmentary  mental  models. 

4.  Reductive  explanation  reinforces  and  preserves  itself  through  misconception  networks 
and  through  knowledge  shields. 

5.  Knowledge  shields  lead  to  wrong  diagnoses  and  enable  the  discounting  of  evidence. 
Therefore  learning  must  also  involve  unlearning  and  relearning. 

Therefore,  advanced  learning  is  promoted  by  emphasizing  multiple  interconnectedness  of  cases 
and  concepts  along  multiple  conceptual  dimensions,  and  the  use  of  multiple  representations. 


CFT  emphasizes  the  achievement  of  flexibility  whereas  CTT  emphasizes  the  need  for  unlearning 
and  re-learning.  Both  theories  regard  advanced  learning  as  a  form  of  sensemaking  (discovery, 
reflection)  and  both  regard  learning  as  discontinuous;  advancing  when  flawed  mental  models  are 
replaced,  stable  when  a  model  is  refined  and  gets  harder  to  disconfirm. 

Both  theories  assert  that  certain  forms  of  training  are  not  appropriate  for  advanced  learning — 
instruction  via  compartmentalized  knowledge,  clear  instances,  and  reproductive  memory  tasks 
will  not  promote  advanced  learning.  Learning  at  beginning  and  intermediate  levels  can  certainly 
benefit  from  this  form  of  instruction.  Anderson’s  ACT-R  proposes  that  automatization  of  lower- 
level  learning  facilitates  subsequent  higher-level  learning  (Anderson,  1996;  Anderson,  et  ah, 

2004).  But  for  advanced  learning,  simplification  methods  can  be  counterproductive. 


Both  CFT  and  CTT  theories  caution  against  the  use  of  extrinsic  immediate  outcome  feedback  for 
advanced  learning  because  at  advanced  levels  immediate  extrinsic  feedback  can  prevent  skill 
development  and  reduce  transfer.  At  the  same  time,  both  theories  are  based  on  the  observation 
that  continuous  outcome  monitoring — intrinsic  immediate  outcome  feedback — is  be  critical. 

Both  CFT  and  CTT  theories  note  the  value  of  mentoring,  i.e.,  active  learning  plus  opportunistic 
guidance.  Table  4.6  presents  a  set  of  propositions  concerning  training  that  is  generated  by  a 
merger  of  theories.  These  prescriptions  should  be  understood  in  light  of  the  fact  that  "Training 
for  complex  tasks  is  itself  a  complex  task,  and  most  principles  for  good  instruction  are 
contextual,  not  universal"  (Reigeluth,  personal  communication). 
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Table  4.6  Implications  of  the  merged  Flexibility/Transformation  theory  for  training  and 
instructional  design. 


The  Good  Mentor 

Uses  a  modeling  approach  where  appropriate  actions  and  decisions  are  illustrated  in  the  context 
of  the  actual  operational  environment,  and,  when  warranted,  the  learner  is  guided  in  his/her  own 
performance. 

Provides  support  for  management  of  increased  uncertainty  and  cognitive  load  when  the  student 
is  dealing  with  complexity. 

Provides  trainees  at  junior  proficiency  levels  with  particular  help  in  forming  their  initial  mental 
models,  e.g.,  by  promoting  information  seeking  strategies. 

Is  able  to  discern  the  trainee’s  mental  model  and  diagnose  of  the  reason  for  the  trainee’s  poor 
performance.  From  this,  the  mentor  can  predict  when  and  why  the  learner  will  form  a  simplistic 
or  inaccurate  understanding. 

Is  able  to  craft  cases  or  problems  that  deliberately  push  the  trainee  to  the  next  level  of 
understanding  or  reasoning  .skill. 

Presents  scenarios  that  encourage  rethinking  of  concepts  and  mental  models  by  triggering  a 
cognitive  conflict  in  which  some  baffling  event  or  anomaly  triggers  a  loss  of  faith. 

Good  Instruction 

Presents  content  knowledge  and  case  material  using  multiple  representations,  especially  for  ill- 
structured  problems  or  domains. 

Presents  practice  opportunities  in  environments  that  replicate  the  operational  environment  to  the 
greatest  extent  feasible. 

Involves  analysis  of  individual  cases  at  different  times,  in  different  contexts,  for  different 
purposes,  using  multiple  representation,  and  from  different  perspectives  is  essential  for 
advanced  knowledge  acquisition. 

The  apprentice  works  with  the  mentor  to  solve  problems,  while  also  making  explicit  the 
strategies  being  adopted  as  well  as  the  apprentice’s  knowledge  and  reasoning. 

Covers  content  more  than  once  to  insure  learning  of  case  complexity,  conceptual  complexity 
and  context-dependent  variability. 

Uses  feedback  that  informs  the  trainee  of  what  to  do  differently  (process  feedback  versus 
outcome  feedback). 

Scaffolds  the  trainee  to  engage  in  sensemaking  and  thereby  generate  their  own  feedback. 

Scaffolds  the  trainee  in  going  from  recognition  of  a  flawed  mental  model  by  making  a  richer 
model. 

Involves  instruction  and  practice  in  handing  ill-structured  problems,  that  is,  problems  in  which 
goals  are  unclear  or  morph. 

Instructs  in  variability  using  extreme  cases,  and  non-routine  cases  illustrating  context- 
dependence  of  concepts  and  patterns.  (With  the  caution  that  the  use  of  extreme  cases  can  be 
counterproductive  at  intermediate  stages  of  learning.) 

Emphasizes  multiple  interconnectedness  of  cases  and  concepts  along  multiple  conceptual 
dimensions. 
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While  useful,  CFT  and  CTT  do  not  address  all  aspeets  of  learning,  and  not  even  all  aspeets  of 
learning  of  complex  tasks.  An  even  more  expansive  view  is  needed  for  designing  training  based 
on  incremental  complexification  of  tasks,  and  identifying  "desirable  difficulties"  as  a  function  of 
domain  and  learning  level.  For  example,  the  Simplifying  Conditions  Method  (Reigeluth,  1999) 
can  be  explored  for  determining  perceived  difficulty  of  cases  for  and  sequencing  them 
appropriately.  It  will  be  necessary  to  explore,  for  each  domain,  the  context-dependence  of  most 
methods  for  fostering  learning,  as  a  function  of  instructional  level,  especially  the  requisite  levels 
of  fidelity  and  authenticity. 

4.3.2.  Finding  the  Right  Training  and  Mentoring  Methods 

Discussions  at  the  Working  Meeting  of  training  methods  focused  on  feedback,  retention, 
transfer,  and  mentoring. 

Feedback 


With  regard  to  feedback,  we  know  of  the  substantively  different  kinds  (outcome,  process, 
immediate,  delayed,  etc.)  but  cannot  make  broad  generalizations.  Feedback  is  not  always  helpful. 
It  is  sometimes  helpful  in  the  short-term  but  not  in  the  log  term.  And  so  forth.  It  may  be  possible 
to  map  feedback  regimens  onto  skills/tasks  at  least  to  some  extent.  Thus,  for  example,  jobs/tasks 
that  involve  difficult  critical  thinking  and  sensemaking  might  benefit  from  delayed  process 
feedback.  Using  the  Pareto  Principle  it  would  be  possible  to  possible  to  formulate  guidance  in  the 
use  of  feedback 

•  Remembering  Reigeluth's  cautionary  tale  that  "Training  for  complex  tasks  is  itself  a 
complex  task,  and  most  principles  for  good  instruction  are  contextual,  not  universal," 
training  would  certainly  benefit  from  a  more  extensive  empirical  base  to  guide  the 
integration  of  different  types  of  feedback  into  instruction  at  different  levels. 

Retention  and  Decay 

We  know  about  the  general  course  of  forgetting,  ranging  from  minutes  (lab  science)  to  years,  and 
for  various  sorts  of  learning  materials.  There  is  consensus  on  the  claim  that  the  level  of 
proficiency  or  overlearning  achieved  during  initial  acquisition  is  the  best  predictor  of  skill  decay. 
With  regard  to  skill  decay  and  “types”  of  knowledge,  some  assert  that  declarative  knowledge  is 
less  often  forgotten  and  procedural  knowledge  is  more  often  forgotten;  whereas  others  assert  that 
procedural  knowledge,  once  overlearned,  is  highly  resistant  to  decay,  as  in  riding  a  bicycle. 
Healy’s  Procedural  Reinstatement  Principle  (2007)  is  based  on  the  premise  that  procedural 
knowledge  (i.e.,  skill)  is  more  durable  than  declarative  knowledge. 

Decay  differs  for  different  tasks.  Currently,  decisions  about  retraining  are  often  made  by 
negotiation.  So  in  piloting,  for  instance,  a  hiatus  of  some  length  might  be  traditionally  mapped 
onto  a  retraining  course  of  20  sorties,  but  there  may  only  be  funding  for  10,  so  a  compromise  of 
15  is  achieved  by  negotiation. 


142 


Report  on  FA8650-09-2-6033 


•  We  currently  have  no  useful  empirical  base  about  individual  difference  predictors  of 
retention,  such  as  age,  gender,  cognitive  ability  level,  education  level,  etc. 

•  We  currently  have  no  useful  empirical  base  about  how  periodic  booster  sessions  (number, 
spacing)  affect  learning  and  forgetting  rates. 

•  We  currently  have  no  useful  empirical  base  that  informs  about  differential  decay  of 
particular  skills.  Booster  training  should  focus  on  refresh  of  skills  that  would  ordinarily 
show  faster  drop-off. 

An  important  aspect  of  retention  of  skill  is  that  following  hiatus  the  work  will  have  changed. 
Retention  is  not  simply  gaining  back  skills  but  adapting  to  new  world.  Thus,  re-leaming  must  be 
an  integral  part  of  post-hiatus  training  or  refresher  instruction. 

The  issues  and  challenges  in  the  military  of  skill  decay  due  to  hiatus  assignments,  are  by  no 
means  unique  to  the  military.  The  example  was  given  of  corporate  practice.  CEOs  and  senior 
managers  get  cut  off  from  experiences  that  keep  them  fresh,  and  then  they  make  bad  decisions. 
Steve  Jobs  is  a  counter-example,  we  might  say,  illustrating  accelerated  updating  more  than 
accelerated  learning  since  people  aren’t  learning  from  scratch.  Some  notion  of  “accelerated 
updating”  should  be  added  to  the  list  of  kinds  or  definitions  of  acceleration. 

The  opportunity  and  need  is  to  actually  track  changes  in  job  requirements  longitudinally  in  one 
or  more  selected  domains  where  there  is  a  rapidly  changing  job  environment  (e.g.,  technology 
adoption  issues).  This  would  be  especially  pertinent  with  regard  to  the  aging  civilian  population. 
It  might  be  possible,  and  would  certainly  be  useful,  to  developing  measures  of  motivation  that 
might  predict  attrition. 

•  We  currently  have  no  useful  empirical  base  about  adaptation  retraining. 

•  We  currently  have  no  useful  empirical  base  about  differential  decay  of  “types”  of 
knowledge  or  skill  in  the  context  of  complex  work  in  professional  domains,  on  the 
assumption  that  transfer  is  either  between  problem  types  or  between  sub-domains  within 
the  broader  domain. 

We  have  presented  the  generalization  that  decay  is  less  for  skill  learning  than  knowledge 
learning.  The  fuller  generalization  is  that  for  skill  learning,  retention  is  strong  but  transfer  is 
limited,  whereas  for  fact  learning,  retention  is  poor  but  transfer  is  robust.  This  leads  directly  to 
the  topic  of  transfer. 

Transfer 


Principles  of  transfer  from  academic/educational  research  provide  some  guidance,  but 
simultaneously  point  to  gaps  in  our  empirical  base.  This  is  detailed  in  Table  4.7.  (This  does  not 
include  the  ’’functional  task  principle,”  for  reasons  explained  above  in  section  4.2.) 
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Table  4.7  Principles  of  training  and  transfer  from  academic/educational  research  and 
opportunities  for  empirical  research. 


Principle 

Opportunities 

Specificitv  of  Training  Principle 

For  optimal  performance,  the  entire  configuration  of  task 
requirements  during  training  needs  to  match  those  in  the  field 
as  closely  as  practically  possible.  Retention  and  transfer  are 
reduced  when  conditions  of  learning  differ  from  those  during 
subsequent  testing  (Healy,  et  ah,  2006). 

This  principle  has  in  many  cases 
already  been  mapped  onto 
training  programs.  It  likely 
applies  broadly  to  skills, 
knowledge  and  the  achievement 
of  proficiency. 

Contextual  Interference  Principle 

Interference  during  learning  facilitates  later  retention  and 
transfer  (Battig,  1979;  Schneider,  et  ah,  1995). 

Determining  domain  or  content- 
appropriate  specification  on 
what  counts  as  “interference.” 

Training  Difficultv  Principle 

Any  condition  that  causes  a  desirable  difficulty  during 
learning  facilitates  later  retention  and  transfer  (Bjork,  1994; 
McDaniel  &  Einstein,  2005;  Schneider,  et  ah,  2002). 

Determining  domain  or  content- 
appropriate  specification  on 
what  counts  as  “desirable 
difficulty.” 

Procedural  Reinstatement  Principle 

Specificity  (limited  transfer)  occurs  for  tasks  based  primarily 
on  procedural  knowledge  (i.e.,  skill)  whereas  generality 
(robust  transfer)  occurs  for  tasks  based  primarily  on 
declarative  information,  or  facts. 

Determining  sensible 

distinctions  on  “types”  of 
knowledge,  as  appropriate  to 
individual  domains. 

Confirming  the  reinstatement 
effect  for  professional  domains. 

With  the  exception  of  the  Specificity  of  Training  Principle  (as  noted  above  in  the  left-hand 
column  of  Table  4.7),  the  generalizability  of  these  principles — carrying  them  over  from 
academic/educational  research  into  domains  of  professional  proficiency  and  complex  cognitive 
work — is  not  at  all  clear.  A  principle  reason  is  related  to  the  relative  simplicity  of  laboratory 
tasks  and  the  relative  complexity  of  work  domains.  Training  and  transfer  contexts  in 
academic/educational  research  are  highly  similar  and  what  is  changed  is  one  or  more  details  of 
individual,  isolatable  tasks.  In  the  work  domain  context,  what  is  the  transfer  from  and  what  is  it 
to? 

Transfer  is  traditionally  thought  of  in  terms  of  gains  (or  losses)  in  performance  that  is  easily 
measured  (e.g.,  correct  responses  or  accuracy  over  time)  on  isolated  fixed  tasks  (historically, 
typing,  pursuit  rotor,  etc.).  This  concept  is  of  limited  use  in  the  analysis  of  complex  domains, 
complex  dynamic  jobs,  and  complex  cognitive  work.  Researchers  today  speak  of  notions  such  as 
“deep  learning”  of  concepts,  and  the  ability  to  make  meaningful  connections  between  previous 
learning  and  novel  situations.  The  study  of  transfer,  and  especially  applications  to  the  analysis  or 
complex  work  domains,  needs  to  break  out  from  this  tradition  of  transfer  of  fixed  skill  or  fixed 
knowledge. 

One  would  not  expect  an  expert  at  urban  firefighting  to  have  all  the  reasoning  capabilities  of  a 
firefighter  experienced  in  wildland  fires.  Yet,  both  are  firefighters  and  they  engage  in  many  of 
the  same  “tasks.”  One  would  not  expect  an  expert  at  forecasting  severe  storms  in  the 
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southeastern  US  to  be  competent  at  predicting  severe  weather  in  the  Canadian  Rockies,  though 
both  involve  the  same  data  types,  the  same  principles  of  atmospheric  dynamics,  and  so  forth.  So, 
the  issue  is,  what  are  the  bounds? 

What  this  discussion  shows  is  that  generalizations  about  transfer  are  difficult  to  make,  and  the 
default  assumption  is  that  knowledge  and  skill  do  not  transfer  (i.e.,  expertise  is  highly  domain- 
specific).  On  the  other  hand,  there  is  the  paradox  of  domain  specificity  (Table  4.2,  above) 
(Expertise  is  defined  by  experience  at  coping  with  diverse  situations  that  stretch  the  competence 
envelope,  so  there  must  be  transfer.) 

What  is  needed  is  a  conceptual  designation  other  than  the  word  “transfer”  to  capture  the 
phenomenon  at  hand.  “Transposition”  might  be  an  appropriate  term.  The  skill  or  knowledge  or 
capacity  that  is  learned  in  one  application  but  subsequently  utilized  in  some  other  context  is  not 
adapted  or  mapped  literally.  Rather,  it  undergoes  some  sort  of  modification  or  transformation  or 
realignment. 

For  application  to  cognitive  work  and  complex  domains,  the  assumption  must  be  that  transfer  of 
capability  (competence  and  proficiency)  is  desired  when  the  transfer  is  from  one  problem  type  to 
some  other  substantively  different  problem  type  within  a  single  sub-domain,  or  when  the  transfer 
is  between  problems  in  different  subdomains. 

With  regard  to  transfer,  we  do  not  have  a  useful  empirical  base  about  job  requirement  changes 
and  transfer  of  skills  within  careers  (e.g.,  piloting  a  jet  versus  a  UAV;  Air  Traffic  Control  with 
changing  technology)  and  knowledge  obsolescence  in  jobs  and  what  motivates  people  to  engage 
in  the  type  of  deliberate  practice  needed  to  maintain  and  develop  skill.  As  one  Working  Meeting 
participant  put  it, 

“The  issue  is  transfer  of  what  to  what.  We  need  to  define  all  the 
variables  or  dimensions  in  the  transfer.  We  need  operational 
definitions  so  that  we  can  measure.  It  is  really  tough  nut  to  crack.  ” 

Mentoring 

A  number  of  Working  Meeting  participants  discussed  issues  and  empirical  gaps  on  topics  of 
mentoring. 

We  may  not  be  giving  learners  the  best  opportunity  to  become  experts  if  we  don’t  also  train 
mentors  and  experts  in  helping  that  person  learn.  It  seems  like  many  mentorship  programs  rely 
on  a  “watch  and  learn”  mentality  where  the  deeper  knowledge  of  the  mentor  may  not  be  obvious. 
Shared  experience  is  that  experts  do  not  necessarily  make  good  mentors.  The  most  experienced 
and  knowledgeable  workers,  who  are  regarded  as  experts  because  they  make  the  job  look  easy, 
often  do  a  terrible  job  of  describing  what  it  is  they  do  and  know.  Thus  there  has  emerged  the 
concept  of  the  “expert  mentor.”  While  we  have  some  ideas  of  what  makes  for  a  good  mentor  (see 
Table  4.6,  above), 

•  We  currently  have  no  useful  empirical  base  on  how  to  identify  individuals  who  would  or 
could  become  good  mentors. 

•  We  currently  have  no  useful  empirical  base  on  how  to  teach  people  how  to  be  mentors. 
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•  We  currently  have  no  useful  empirical  base  on  specific  mentoring  techniques  and  tools 
that  mentors  can  use  to  share  their  expertise  with  mentees. 

•  We  currently  have  no  useful  empirical  base  on  such  factors  as  cost-benefits  of  mentoring, 
the  impact  of  organizational  culture,  effects  of  extrinsic  motivation  (e.g.,  rewards)  for 
mentoring,  etc.  (Should  mentors  be  rewarded  based  on  how  well  their  mentees  perform?) 

•  Do  we  reward  /  recognize  people  for  the  mentoring  aspects  of  their  work?  That  is,  do  we 
reward  them  based  on  how  well  their  mentees  perform? 

While  there  is  some  empirical  evidence  that  engaging  trainees  who  are  relatively  more  advanced 
in  the  tutoring  of  trainees  who  are  relatively  less  advanced,  the  positive  findings  in 
academic/educational  research  have  no  counterpart  in  studies  of  “real  world”  job  contexts.  Is 
there  value  in  teaching  novices  how  to  be  mentors?  Nor  do  we  have  a  handle  on  the  effects  of 
tutoring  on  the  tutors.  Specifically,  what  is  the  benefit  of  practicing  trainees  from  the  very  start  in 
how  to  describe  the  skills  they  are  learning  (knowledge  sharing  skills)  at  the  same  time  that  they 
are  practicing  the  skills  in  which  they  will  become  experts? 

Final  Comment 


It  was  noted  a  number  of  times  in  the  Working  Meeting  that  many  of  the  notions  that  were 
presented  and  discussed  entail  a  decomposition  of  domains,  jobs,  and  tasks.  The  ideal  would  be  a 
decomposition  in  terms  of  competencies  that  can  be  mapped  on  to  decay  rates  and  thence  onto 
retraining  plans.  Decomposition  would  be  with  respect  to  types  of  learning  required,  adaptation 
to  different  learning  styles,  and  so  forth.  There  are  certainly  significant  theoretical  and 
methodological  issues  with  such  a  traditional  human  factors  approach.  As  has  been  mentioned, 
the  notion  of  a  “task”  has  limited  utility  and  applicability  in  the  analysis  of  complex  cognitive 
work.  If  one  takes  an  alternative  approach  of  analysis  in  terms  of  the  primary  macrocognitive 
functions,  one  is  left  with  a  roster  of  very  high-level  capacities  (e.g.,  sensemaking,  resilience, 
mental  model  formation,  etc.)  that  cannot  easily  be  mapped  onto  instructional  designs.  Domains 
and  jobs  are  certainly  multidimensional  (not  to  mention  that  they  are  moving  targets).  This 
presents  a  significant  conceptual  and  methodological  challenge: 

•  We  currently  have  no  useful  empirical  base  on  alternative  schemes  for  mapping  of 

domains  with  respect  to  (re)training. 

4.3.3  Methods  and  Designs  for  Studying  Acceieration 

In  this  Section,  which  summarizes  the  Working  Meeting,  we  note  only  the  most  salient  ideas  that 
were  raised  about  acceleration  methods,  experimental  designs,  and  research  plans.  The  full 
integration  of  ideas  and  recommendations  concerning  acceleration  is  presented  in  Section  6  of 
this  Report,  merging  ideas  about  acceleration  from  the  literature  review  and  the  Working 
Meeting. 

Acceleration 


Ideas  presented  at  the  working  Meeting  pointed  to  each  of  the  four  main  meanings  of 
“accelerated  proficiency”  that  were  presented  in  Section  1,  above:  Rapidizided  Training, 
Rapidized  Knowledge  Sharing,  Accelerated  Proficiency,  and  Facilitated  Retention.  A  nuance  on 
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Facilitated  Retention  emerged  at  the  Working  Meeting,  that  of  “Accelerated  Updating.”  This  is 
the  notion  that  hiatus  maintenance  training  and  post-maintenance  Refreshing  should  focus  on 
aspects  of  the  work  that  have  changed  during  the  hiatus.  This  notion  is  predicated  on  the 
“moving  target”  principle  that  governs  sociotechnical  workplaces. 

Some  of  the  suggestions  for  acceleration  methods  cut  across  these  kinds  of  acceleration.  For 
example,  achieving  accelerated  proficiency  by  better  training  of  the  trainers  relates  to  ideas  of 
accelerated  proficiency  and  also  rapidized  training  (i.e.,  improved  mentoring). 

"Just-In-Time”  training  (JIT)  was  raised  as  an  example  of  accelerated  learning.  The  assumption 
of  JIT  is  that  if  just  enough  information  can  be  provided  quickly,  in  the  work  setting,  that  this 
would  make  it  possible  to  preserve  the  capability  of  the  work  system.  While  JIT  is  a  form  of  on- 
the-job  training,  the  intent  is  to  cope  with  rapid  change.  In  JIT,  short  slices  of  instruction  (<  5 
minutes)  are  targeted  to  just  one  or  a  few  concepts  or  processes.  Different  from  classroom 
instruction,  the  JIT  material  can  be  delivered  online,  and  can  use  enriching  multimedia  such  as 
video  with  narration.  JIT  as  defined  depends  on  some  methods  for  Just-in-Time  Assessment, 
allowing  training  managers  to  rapidly  estimate  what  the  trainee  has  learned  or  what  skills  have 
been  acquired. 

One  view  is  that  JIT  is  necessary  because  of  the  paradox  of  limited  time  in  the  schoolhouse 
versus  the  constantly  changing  nature  of  work  (see  Section  4.2,  above).  But  the  concern 
expressed  at  the  Working  Meeting  is  that  JIT  cuts  against  what  is  known  about  the  development 
of  proficiency,  and  learning  in  general.  Specifically,  the  acquisition  of  proficiency  depends  on 
practice,  practice,  and  more  practice.  On  this  view,  just-in-time  training  is  a  recipe  for  just-in- 
time  disaster  especially  since  proficiency  may  be  most  urgently  needed  in  pressing 
circumstances. 

Scenario-Based  Training 

A  main  theme  at  the  Working  Meeting  was  training  and  accelerated  learning  by  means  of  cases 
or  scenarios.  This  notion  is  entailed  by  the  law  of  frequency,  the  principle  of  domain  specificity, 
and  is  supported  by  both  research  and  practical  experience.  Several  aspects  of  the  development 
of  proficiency  (discrimination,  transfer,  progressive,  tacit  knowledge,  etc.)  point  to  the  use  of 
multiple  scenarios  or  cases  in  training.  These  would  be  sequenced,  according  to  the  notion  of 
“increasingly  complex  microworlds”  (Burton,  Brown,  &  Fischer,  1984;  Bunderson,  2006).  Both 
contrast  and  repetition  effects  are  possible  even  by  changing  only  a  few  problem  variables  of 
interest. 

Exemplars  of  successful  acceleration  are  Top  Gun,  the  National  Training  Center,  and  the  Think 
Like  a  Commander  Program.  Shared  characteristics  of  these  programs  are  presented  in  Table  4.8. 
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Table  4.8  Characteristics  of  effective  scenario-based  training. 


•  The  scenarios  are  tailored  to  learners  (individual  and/or  group),  depending  on  level  of 

achievement,  preparedness,  or  other  factors. 

•  Scenarios  are  created  from  lessons  learned. 

•  Scenarios  are  created  based  on  empirical  knowledge  on  how  highly  proficient 

workers  apprehend  problems. 

•  Scenario  training  assumes  high  intrinsic  motivation  of  the  trainee  to  work  hard,  on 

hard  problems. 

•  For  any  given  level  of  training,  the  scenarios  are  tough.  They  are  novel  to  the  learner, 

challenging  them  in  ways  described  by  Cognitive  Flexibility  Theory  and 
Cognitive  Transformation  Theory  (see  above). 

•  Typically  there  is  some  adversary,  such  as  a  superior  or  more  capable  opposing  agent 

or  force. 

•  The  trainee  is  challenged  to  learn  to  think  like  the  adversary. 

•  The  fidelity  is  as  high  as  needed.  (In  order:  desktop  exercises  using  paper  and  pen, 

virtual  worlds  presented  on  computer  monitors,  virtual  environments,  very  high 
fidelity  simulators,  simulated  villages.) 

•  Scenarios  mimic  the  operational  context. 

•  There  is  a  designed-in  ability  for  observers  to  measure  what  happened. 

•  The  observers  are  experts. 

•  There  are  multiple  reviews,  not  one  single  “after  action”  review. 

•  Reviews  provide  outcome  and  process  feedback. 

•  Reviews  include  retrospection  and  the  analysis  of  decision  processes,  emotional  state 

of  mind,  teamwork,  mental  projection  to  the  future,  and  other  macrocognitive 
processes. 

•  The  goal  is  for  trainees  to  acquire  strategic  knowledge,  adaptability,  and  resilience. 


We  have  emphasized  simulation-based  scenarios,  but  the  training  at  all  levels  of  proficiency  will 
always  use  some  appropriate  mix  of  training/assessment  environments.  Much  practice,  including 
advanced  practice,  can  be  done  cost-effectively  in  a  classroom  and/or  virtual  environment. 
Validation  of  achieved  level  of  proficiency  should  ideally  be  conducted  in  virtual  or  near- live 
environment,  in  a  reliable,  valid  scenario. 

Research  Designs.  Plans,  and  Requirements 

Certain  kinds  of  studies  could  commence  immediately  once  domains  are  selected.  Archival 
research  could  begin  the  process  of  forming  a  proficiency  scale,  forming  good  measures,  and 
building  a  corpus  of  cases.  One  could  find  a  cohort  and  look  back  over  their  record. 
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There  was  a  consensus  at  the  Working  Meeting  that  a  program  of  research  on  acceleration  should 
involve  the  study  of  at  least  two  interestingly  and  substantively  different  domains. 

There  was  a  consensus  at  the  Working  Meeting  that  research  on  acceleration  will  necessitate  the 
application  of  proficiency  scaling  methodology.  This  includes  career  analysis,  sociometric 
analysis,  and  performance  testing  (see  Hoffman,  2010).  This  methodology  is  well  understood. 
However,  proficiency  scales  have  to  be  domain- appropriate.  Currently,  we  lack  validated 
proficiency  scales  for  many  jobs/tasks. 

It  was  pointed  out  that  we  have  sophisticated  tools  for  analyzing  developmental  studies  (e.g., 
longitudinal  data  sets)  such  as  multi-level  modeling  techniques  and  trajectory  analysis 
techniques.  At  the  same  time,  it  was  pointed  out  that  the  research  will  have  to  commence  before 
“good”  measures  are  devised  and  validated.  A  primary  purpose  of  measurement  is  always  to 
improve  the  measures.  Traditional  human  performance  measures  will  be  inadequate  for  the 
analysis  of  complex  cognitive  work  (Hoffman,  2010).  In  addition,  measurement  methodology  for 
the  study  of  acceleration  will  itself  have  to  adapt  over  the  course  of  the  research  since  the  study 
will  be  of  “moving  target”  domains. 

It  would  be  a  major  project  to  develop  and  compare  different  training  techniques  (hands-on, 
simulation  environments  of  varying  fidelity)  and  conduct  longitudinal  studies  within  a 
population  of  learners  that  represent  a  spectrum  of  proficiency  levels.  It  would  be  a  major  project 
to  conduct  such  study  long  enough  (more  than  five  years,  realistically)  to  cover  hiatus  periods 
over  actual  careers.  Nevertheless,  it  is  precisely  such  study  that  would  provide  the  necessary 
empirical  base  (see  the  bullet  lists  in  Section  4.3.2,  above). 

Rather  than  tracking  a  group  from  an  apprenticeship  all  the  way  to  high  proficiency,  one  can 
conduct  cohort-select  designs.  There  are  multiple  groups  of  trainees  who  represent  different 
levels  of  current  proficiency.  Levels  could  be  any  or  all  of:  senior  apprentice,  junior  journeyman, 
journeyman,  senior  journeyman,  junior  expert.  Once  the  “grain”  is  determined,  the  groups  could 
each  be  tracked  over  time. 

In  addition,  a  preliminary  process  of  constructing  the  domain-appropriate  proficiency  scale, 
would  involve  a  detailed  retrospective  analysis  of  each  trainee’s  experiences  and  training. 
Proficiency  scaling  methodology  is  reasonably  well  understood.  One  would  identify  individuals 
who  learn  fast  and  slow  and  conduct  retrospective  analyses  to  find  out  why.  One  could  identify 
the  top  performers  at  each  proficiency  level  (test  scores,  records  of  past  performance  during 
training)  and  work  backwards  to  find  the  success  factors.  One  could  study  acquisition 
retrospectively  and  study  degradation  and  retraining  prospectively. 

Thus,  in  such  designs  one  could  examine  development  both  retrospectively  and  prospectively 
and  thereby  “cover”  more  than  the  five  to  eight  year  duration  of  the  actual  research  project  itself. 
Several  capabilities  will  be  needed  to  support  scenario  design,  beyond  access  to  highly 
experienced  domain  practitioners  and  all  the  pertinent  “lessons  learned.”  Specifically,  scenarios 
may  have  to  be  parameterized,  so  that  they  can  be  rapidly  or  even  dynamically  constructed.  An 
instructor  might  introduce  “twists”  into  on-going  scenarios  to  address  proximal  learning 
objectives  (e.g.,  triggered  by  an  observation  of  biased  reasoning). 
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Achieving  that  will  itself  require  a  research  program  to  baseline  scenarios  for  initial  assessment 
against  learning  objectives  and  to  build  a  corpus  of  common  and  rare  scenarios  for  learning 
generalized  and  specialized  knowledge  and  procedures. 

Several  classes  of  technology  may  be  central  to  implementing  accelerated  learning  concepts: 

•  Pervasive  technology  for  human  performance  measurement  is  needed  to  estimate  the 
effects  of  training,  in  the  short  term,  and  to  validate  and  calibrate  the  impact  of 
competencies  on  task  performance,  in  the  long  term. 

•  Scenario  engineering  technology  is  required  to  dynamically  adapt  training  simulations 
and  games  in  ways  that  ensure  students  are  continuously  stimulated  by  events  that 
exercise  and  extend  their  competencies.  In  the  long  term,  such  technology  should  help  us 
to  assess  whether  training  in  specific  competencies  is  feasible  in  simulations. 

•  Computational  or  mathematical  models  of  competency  and  its  impact  on  performance 
(e.g.,  accelerating  performance,  raising  quality)  are  needed,  or  will  be  needed  eventually 
to  move  from  empirical  study  to  model-based  design  that  can  support  the  performance  of 
military  organizations  under  different  assumptions  about  staff  composition  and  size. 

The  Working  Meeting  participants  recognized  that  the  goal  of  training — for  trainees  to  acquire 
strategic  knowledge,  adaptability,  and  resilience — and  the  goal  of  accelerating  that  acquisition — 
are  daunting  from  a  scientific  point  of  view.  Nevertheless,  the  consensus  was  that  the  research  is 
possible  given  what  is  presently  known  empirically  and  about  methodology.  As  one  Working 
Meeting  participant  put  it, 

'"Skills  proficiency  is  attained  through  scenario-based  acquisition 
of  the  skill  to  some  criterion,  more  practice  after  achieving  that 
criterion  and  then  validation  of  proficiency  through  assessment  of 
performance.  Proficiency  can  be  attained  with  good  training 
design  and  situated  learning  in  a  short  time.  Both  skill  retention 
and  the  achievement  of  higher  levels  of  proficiency  are  achievable 
through  the  same  sequence  of  acquisition,  practice,  validation 
though  with  fewer  scenarios  needed.  ” 
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5.0  CONCLUSIONS  AND  RECOMMENDATIONS 

In  his  summary  of  results  from  the  two  Accelerated  Learning  meetings,  Andrews  (2009)  listed 
the  following  key  science  and  technology  challenges  for  accelerated  learning: 

•  Understanding  the  requirements  for  high  proficiency  and  how  to  accelerate  its  attainment. 

•  Understanding  how  to  increase  retention  of  high  proficiency  (facilitated  retention). 

•  Development  of  serious  games  that  can  accelerate  the  achievement  of  high  proficiency. 

•  Development  of  "rapid  cognitive  task  analysis"  methods  to  understand  training  problems. 

•  Development  of  career  path  methods  to  facilitate  progress  along  the  proficiency 
continuum  and  the  achievement  and  retention  of  high  proficiency. 

•  Calculating  the  costs  and  benefits  of  instituting  accelerated  learning  methods,  and  the 
risks  of  not  doing  so. 

The  intent  of  this  section  of  this  Report  is  to  take  these  ideas  a  step  further  by  suggesting 
elements  of  a  roadmap  for  research  on  accelerated  proficiency  and  facilitated  retention:. 
Suggestions  for  acceleration  methods  include  leveraging  the  important  features  of  practice  and 
mentoring,  the  technologies  of  virtual  reality  training,  multi-media  for  training  at  complex  tasks, 
and  leveraging  computer  technology  to  develop  libraries  of  representative  and  tough  tasks  and 
knowledge  bases  that  capture  expert  knowledge  and  .skill. 

5.1.  Interdisciplinarity 

Research  on  acceleration,  in  any  of  its  four  senses,  will  require  a  multidisciplinary  team. 

•  Researchers  who  study  expertise  understand  the  nature,  development,  and  assessment  of 
proficient  performance.  They  also  command  methods  for  the  elicitation,  representation, 
archiving,  and  transmission  of  knowledge  and  skill.  They  understand  what  it  is  that  the 
project  is  trying  to  accelerate. 

•  Researchers  who  conduct  large-scale  longitudinal  studies  understand  and  have  experience 
at  tracking  learning  and  skill  changes  and  their  components  across  extended  periods  of 
time.  They  understand  the  practicalities,  measurement  issues,  and  control  requirements 
for  such  large-scale  studies. 

•  Researchers  who  study  the  fundamental  processes  of  learning  and  memory,  in 
experimental  psychology’s  laboratory  studies  reveal  the  main  variables  that  influence  the 
acquisition,  retention,  and  use  of  knowledge.  Methods  of  controlled  study  of  transfer  and 
flexibility  of  training  are  critical  in  theories  of  learning,  as  are  issues  of  memory  and  skill 
decay. 

•  Trainers  within  an  organization  are  responsible  for  developing  training  materials, 
tracking  training  effectiveness,  etc.  They  know  the  current  methods  of  training  that  are 
used  within  an  organization,  as  well  as  the  organization's  history  and  culture  regarding 
training.  They  have  insights  into  what  works  for  what  kinds  of  skill,  why  it  works,  and 
why  it  sometimes  does  not.  They  have  insights  into  the  interaction  of  training  methods, 
training  materials,  and  the  individual  capabilities  and  learning  styles  of  trainees.  All  these 
considerations  would  be  critical  for  the  design  of  attempts  at  accelerating  the 
achievement  of  proficiency. 
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•  Human  resource  specialists  have  responsibility  for  writing  job  descriptions  and  for 
creating  and  conducting  evaluation  and  promotion  processes.  They  have  experience  with 
the  ways  and  means  by  which  people  are  able  to  gain  entry  to  an  organization  and  the 
extent  to  which  they  subsequently  thrive.  Any  effort  at  accelerating  expertise  will  require 
flexibility  and  creativity  in  matters  of  personnel  and  human  resource  management. 

•  Instructional  design  is  a  designated  work  line  for  creating  and  certifying  instructional 
materials  and  methods.  Efforts  at  expertise  acceleration  will  likely  need  to  operate  within 
such  an  extant  organizational  context  for  conducting  instruction. 

•  Researchers  who  study  team  cognition  understand  organizations  as  systems,  for  example 
in  their  dynamics,  interdependence  of  processes,  emergent  properties,  adaptability,  and  so 
forth.  Any  program  of  expertise  acceleration  will  need  to  take  place  within  an  existing 
organizational  climate  and  structure. 

•  Educational  technologists  provide  tools  for  selective  experience  compaction,  involving 
such  devices  as  simulations,  games,  virtual  reality,  and  other  case-based  educational 
platforms. 

•  Researchers  who  study  self-regulation  provide  understandings  of  the  motivational  and 
emotional  requirements  of  dedicated,  intense  effort,  and  strategies  for  coping  with  failure 
and  disappointment,  and  "perceived  efficacy"  to  maintain  interest  in  the  face  of  inevitable 
periodic  setbacks. 

•  Researchers  who  have  studied  Mentoring  or  Tutoring  are  familiar  with  directed  learning 
activities  associated  with  implicit  job  "know  how"  and  “tricks-of-the  trade.”  Specialists  in 
this  category  understand  apprenticeship  and  other  forms  of  one-on-one,  on-the-job 
learning  and  instruction. 

5.2  Measurement  and  Analysis  Issues 

5.2. 1  Research  Need:  Rapidized  Cognitive  Task  Anaiysis 

Eor  many  military  tasks,  there  is  insufficient  time  available  for  training  prior  to  deployment.  Eor 
such  mission-critical  tasks,  it  will  be  necessary  to  rapidly  conduct  the  cognitive  task  analysis 
required  to  inform  the  creation  of  instructional  materials  and  methods  (e.g.,  serious  games,  tough 
cases,  etc.)  that  are  themselves  required  for  accelerated  learning.  Currently  there  are  no  baseline 
data  on  time  required  for  deep  and  rich  task  analysis  for  tasks  having  high  intrinsic  cognitive 
load.  Thus,  we  have  no  methodology  and  research  base  on  the  question  of  how  cognitive  task 
analysis  itself  might  be  accelerated  (Andrews,  2009).  As  military  missions  and  tasks  continue  to 
evolve  and  change,  sometimes  quickly,  it  is  imperative  that  we  rapidize  the  methodologies  that 
are  used  to  form  training  materials  and  methods. 

5.2.2  Research  Need:  Proficiency  Measurement  and  Seating 

Eor  many  military  domains/jobs,  including  ones  mentioned  in  the  literature  and  discussed  at  the 
Working  Meeting,  continuing  effort  is  needed  to  extend  and  refine  measures  of  performance  and 
achieve  some  level  of  consistency  in  measurement  so  that  a  potentially  vast  pool  of  empirical 
data  becomes  available.  This  is  with  respect  to  measuring  performance  beyond  minima! 
proficiency,  measurement  in  terms  other  than  hours  of  practice,  and  measurement  of 
performance  at  complex  cognitive  work.  In  domains  where  hiatus  experiences  are  likely, 
performance  measurement  needs  to  be  conducted  just  prior  to  the  hiatus  onset  and  just  prior  to 
re- assignment  post-hiatus. 
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Proficiency  at  the  highest  levels  ("senior  expert”)  has  been  referred  to  as  “the  5%-ers”  or  “super¬ 
experts”  (Ziebell,  2009).  Achievement  at  this  level  has  to  do  with  predisposition,  personality, 
curiosity,  drive — all  the  experiential  and  motivational  factors  that  separate  out  the  individuals 
who  become  experts  from  those  who  do  not.  While  researching  the  notion  of  acceleration  to 
highest  levels  is  perhaps  outside  the  scope  of  immediate  needs  and  practical  possibilities, 
measurement  appropriate  to  this  level  is  needed,  if  ideas  of  proficiency  and  expertise  are  to  be 
transported  into  the  context  of  selection.  Across  a  broad  spectrum  of  military  jobs,  these 
discriminatory  aspects  of  proficiency  might  be  measured  in  addition  to  the  current  performance 
measures,  and  individuals  tracked  over  time.  This  would,  ultimately,  generate  an  empirical  base 
and  that  might  be  of  use  in  developing  refined  selection  procedures. 


5.2.3  Measurement  for  Robustness,  Resilience  and  Adaptivity 

With  regard  to  complex  cognitive  work  and  emerging  domains  (e.g.,  cyberwork,  etc.),  there  is  an 
outstanding  need  for  measures  of  cognitive  work  and  “high  -level”  or  macrocognitive  skills  (see 
Hoffman,  2009,  in  press;  Klein,  in  press).  There  is  a  general  concern  with  measuring  decision 
making  in  contexts  requiring  adaptivity  and  resilience  (e.g.,  Dodd,  2009). 

Robustness  is  the  ability  to  maintain  effectiveness  across  a  range  of  tasks,  situations,  and 
conditions.  Resilience  is  the  ability  to  recover  from  a  destabilizing  perturbation  in  the  work  as  it 
attempts  to  reach  its  primary  goals.  Adaptivity  is  the  ability  to  employ  multiple  ways  to  succeed 
and  the  capacity  to  move  seamlessly  between  them.  Measurement  for  resilience  and  adaptability 
must  go  well  beyond  traditional  measures  of  the  performance  of  individuals  (hits,  errors,  rate, 
etc.)  (Alberts  &  Hayes,  2003).  Fundamental  to  the  achievement  of  robustness,  resilience  and 
adaptivity  is  the  opportunity  to  practice  at  problems  that  stretch  current  competency  (Ericsson  & 
Lehmann,  1996;  Feltovich  et  ah,  1997).  The  modem  warfighter  must  acquire  knowledge  and 
reasoning  skills  that  pertain  to  critical  domain  goals  but  which  must  be  exercised  in  widely 
differing  situations  or  contexts. 

5.3.  Factors  Involved  in  the  Choice  of  Domains  for  Possible  Study 

For  projects  to  test  notions  of  accelerated  proficiency  or  facilitated  retention,  the  following 
criteria  are  felt  to  be  necessary: 

1.  Research  should  examine  at  least  two  domains,  which  should  be  substantively  and 
interestingly  different.  The  application  of  a  method  of  acceleration  to  a  single  domain  would 
leave  open  the  important  questions  of  validity  and  reliability,  and  generalizability. 

2.  Research  should  examine  one  or  more  domains  that  are  of  significance  to  the  current  and 
future  operational  needs  of  the  military.  This  criterion  suggests  such  domains  as  “human  terrain” 
intelligence  analysis  and  UAV  operations. 

3.  Research  should  examine  domains  that  are  of  importance  across  the  branches  of  the  military. 
This  criterion  suggests  domains  including  weather  forecasting.  Air  Force  Weather  and  Navy 
Meteorology  and  Oceanography  provide  mission-critical  services  to  Army  and  Joint  operations. 
In  addition.  Air  Force  Weather  already  has  initiatives  that  fit  with  the  notions  and  goals  of 
accelerated  learning  (McNulty,  2005).  Another  example  domain  is  electrical  power  generation 
and  coordination.  This  battalion-level  mission  of  the  Corps  of  Engineers  provides  support  to 
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operations  conducted  by  other  branches,  such  as  emplacement,  airfield  support,  SASO,  and 
others. 

4.  Research  should  examine  domains  that  have  analogs  in  the  civilian  sector.  This  criterion  is 
important  because  it  might  engage  private  sector  partners  who,  like  the  military,  have  issues  of 
workforce,  training,  and  knowledge  management.  For  instance,  over  60  percent  of  the  senior 
engineers  in  the  electric  utilities  in  North  America  are  ready  for  retirement,  mandating  intensive 
knowledge  management  programs  and  a  need  for  fast-ramp  training  so  that  a  next  generation  can 
quickly  achieve  high  levels  of  proficiency. 

5.  The  choice  of  domains  should  give  special  consideration  of  critical  competencies  that 
ordinarily  take  years  to  develop.  Examples  include  cultural  understanding,  understanding  and 
evaluation  of  political,  cultural  and  economic  environments  of  the  battlespace,  military 
implications  of  fused  sensing  and  intelligence,  space  management  for  complex  operations  (i.e., 
avoiding  interference),  deception,  influence,  and  information  operations,  and  campaign 
replanning  (Joint  Chiefs  of  Staff,  2005). 
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Table  5.1.  Considerations  in  domain  choice. 


Domain  Affordances 

The  domain  affords  study  of  accelerated  learning  in  the  sense  of  rapidized  training. 

Domains  in  which  one  might  rapidize  the  transposition  of  lessons  learned  from  the 
battlespace  into  the  training  context. 

The  domain  affords  the  study  of  accelerating  the  achievement  of  high  proficiency. 

The  domain  affords  the  study  of  facilitated  retention. 

The  domain  transposes  readily  into  laboratory  study:  Ease  of  study  (factors  including  data 
access,  availability  of  known  and  valid  measures,  etc.);  some  tasks  that  can  be  readily 
modeled  (e.g.,  using  simple  linear  models). 

Complex  Cognitive  Work 

The  task/job  work  requires  highly  organized  knowledge,  reasoning  skills,  and  motor  or 
procedural  skills  as  well. 

The  job/task  work  involves  coping  with  dynamics  and  complexity. 

Domains  requiring  a  capacity  to  be  adaptive  and  resilient. 

The  work  involves  teaming  of  specialists,  and  requires  expert  management  of  teams. 

Desirable  Practitioner  Characteristics 

The  work  depends  on  skill  at  perceptual  (re)learning,  recognition-primed  decision 
making,  sensemaking,  and  the  other  primary  macrocognitive  functions. 

Achievement  of  proficiency  involves  practice  at  tough  cases. 

Practitioner  motivation  and  affect  play  a  key  role. 

Desirable  Organizational  Characteristics 

Domains  with  recently  transformed  strategic  environment,  where  people  have  more  than 
one  duty  assignment. 

There  is  organizational  support  for  processes  of  achieving  higher  levels  of  proficiency. 

Domains  that  involve  work  that  is  largely  "open  source." 

Many  candidate  domains  were  mentioned  and  discussed:  Accelerated  Learning:  Electric 
utilities/power  generation  and  distribution;  UAV  operation;  Maintenance;  Emergency  response; 
Weather  forecasting;  Cultural  awareness  and  skill;  UAV  operations. 

5.4  Knowledge  Management 

Eor  present  purposes,  we  define  knowledge  management  as  the  process  of  capturing,  preserving 
and  sharing  the  knowledge  and  reasoning  skills  of  experienced  workers.  Knowledge 
management  combines  knowledge  elicitation  methods  with  knowledge  sharing  and  training 
methods. 

There  are  a  number  of  opportunities  for  new  initiatives  in  knowledge  management  on  the  part  of 
the  Department  of  Defense  that  would  contribute  to  accelerated  proficiency.  Eor  instance,  the 
DoD  broadly  does  not  do  a  systematic  job  of  managing  its  expertise  post  retirement  (SES,  etc.). 
This  is  a  missed  opportunity  and  wide  open  for  potentially  valuable  research. 

5.4. 1  Building  Knowledge  Models 

Preservation  of  expert  knowledge  (especially  for  individuals  about  to  retire)  is  a  salient 
consideration  in  current  discussions  of  workforce  issues,  technology  and  the  world  economy. 
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(See  Section  1  of  this  Report.)  The  capture  and  preservation  of  expert  knowledge  is  critical  in 
many  organizations  and  in  entire  sectors  of  the  economy  where  there  is  significant  risk  of  loss  of 
capability. 

There  is  no  doubt  that  a  distinguishing  feature  of  expertise  is  depth  and  extent  of  knowledge.  It 
will  continue  to  be  valuable,  sometimes  critical,  to  elicit  and  preserve  expert  knowledge.  Recent 
IHMC  experience  continues  to  affirm  the  finding  that  senior  experts  possess  significant 
knowledge  for  designing  a  process  rationale,  knowledge  that  is  critical  to  explaining  and  in 
understanding  why  things  are  done  the  way  they  are  done.  Invariably  in  eliciting  this  knowledge, 
it  is  noted  that  someone  with  less  experience  might  have  made  mistakes  were  they  unaware  of 
this  knowledge,  sometimes  called  “tacit  knowledge,”  unaware  because  this  knowledge  is 
possessed  by  unique  individuals  and  is  undocumented. 

While  Concept  Mapping  is  not  the  only  method  for  knowledge  capture,  it  is  highly  efficient  for 
knowledge  capture,  and  can  be  complemented  by  other  methods.  Concept  Mapping  can  capture 
rationale  and  legacy  interactions,  as  has  already  been  demonstrated  in  some  of  the  Concept  Maps 
created  for  various  specializations  in  the  electric  utilities.  It  is  also  necessary  to  reveal  expert 
reasoning  skills  and  strategies.  The  Cognitive  Modeling  Procedure  (see  Crandall,  Klein  & 
Hoffman,  2005;  Klein  &  Hoffman,  2008)  is  a  highly  efficient  method  for  generating  models  of 
practitioner  reasoning  and  revealing  differences  in  reasoning  across  levels  of  proficiency. 

Interestingly,  the  process  of  capturing  expert  knowledge  and  skill  for  the  purpose  of 
preservation,  feeds  directly  into  training,  and  not  just  training  in  general  but  training  with  regard 
to  acceleration.  It  is  this  convergence  of  applications  that  represents  a  significant  leveraging 
opportunity  for  research  on  accelerated  learning. 

5.4.2  Identifying  Difficult  Areas 

To  achieve  focus  on  high-payoff  areas  of  study  that  can  inform  a  research  thrust,  and  to  enable 
the  research  to  get  off  to  a  fast  start,  researchers  would  initiate  a  procedure  that  has  been  used 
successfully  in  research  on  tough  tasks,  for  example,  in  the  practice  of  medicine  (Dawson 
Saunders,  Feltovich,  Coulson,  &  Steward,  1990).  This  involves  structured  interviews  with 
experts  in  which  they  are  queried  about  concepts  in  that  are  both  important  and  particularly 
difficult  to  learn,  understand  well,  and  apply  successful  in  real-world  applications.  These  are  the 
key  target  concepts  within  the  domain  of  practice.  Experience  has  shown  that  experts  can  list  a 
great  many  concepts  and  then  pull  out  ones  that  are  especially  difficult.  Agreement  in  responses 
of  experts  reflects  various  degrees  of  consensus  and  increases  confidence  that  certain  areas  are 
fruitful  ones  for  further  study. 

5.4.3  Creating  Case  Libraries 

It  is  widely  recognized  that  training  must  derive  from  cognitive  task  analysis  and  interviews  with 
domain  experts,  to  identify  critical  decision  points,  aspects  of  uncertainty  and  time  pressure,  etc. 
e.g.,  Dawson  &  Caird-Daley,  2009). 

Arguably,  any  attempt  at  accelerating  the  achievement  of  proficiency  will  depend  in  one  way  or 
another  on  the  creation  of  a  case  corpus.  That  will  provide  an  empirical  base  to  establish 
proficiency  scales  and  will  determine  the  “gold  standards”  for  performance.  Based  on  the 
principle  that  high  proficiency  is  defined  as  the  ability  to  cope  with  difficult  cases  and  the 
empirical  fact  that  difficult  cases  tend  to  be  rare,  then  a  corpus  of  cases  developed  for  the 
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purposes  of  knowledge  management  would  be  the  principal  material  used  in  scenario-based  or 
problem-based  training  experiences. 

The  available  literature  shows  that  is  it  possible,  and  sometimes  fairly  easy,  to  build  a  curriculum 
of  cases.  Experts  and  managers,  working  together,  are  typically  successful  at  identifying  the 
unique  and  important  knowledge  areas  in  which  a  particular  expert  excels.  Likewise,  domain 
practitioners  can  readily  identify  those  important  concepts  in  a  domain  that  seem  to  be  especially 
difficult  for  others  to  fully  comprehend.  Any  training  program  will  have  a  set  of  concepts 
deemed  especially  important  to  master.  There  can  even  be  some  institutional  surveying, 
investigation  to  attempt  to  establish  some  consensus  about  what  these  focal  concepts  might  be 
(Feltovich,  Spiro,  &  Coulson,  1993). 

Methods  of  knowledge  elicitation  and  cognitive  task  analysis  would  be  used  to  generate  cases.  A 
number  of  well-understood  methods  can  be  used,  such  as  the  Critical  Decision  Method  (CDM) 
(Hoffman,  Crandall,  &  Shadbolt,  1998).  This  method  requires  participants  to  retrospectively 
analyze  cases  from  their  own  experience  that  caused  them  great  difficulty  and  stretched  their 
professional  capacity.  The  participating  experts  are  directed  to  identify  cases  in  which  the 
targeted  tough  concepts  played  a  significant  role.  Researchers  would  probe  for  instances  of 
enhanced  immersion,  recovery  from  error,  and  adaptability /flexibility. 

An  important  point  about  training  complex  tasks  is  the  role  of  stories  in  the  process  of 
knowledge  transfer.  Orr  (1996)  found  that  “when  troubleshooting  complex  systems,  technicians 
tell  stories  because  the  hardest  part  of  diagnosis  is  making  sense  out  of  a  fundamentally 
ambiguous  set  of  facts,  and  this  is  done  through  a  narrative  process  to  produce  a  coherent 
account”  (p.  186).  In  relation  to  this,  and  in  the  context  of  the  development  of  training  systems, 
case  libraries,  which  include  expert  stories,  can  be  effective  for  training  technical  problems. 
Training  research  using  such  an  approach  demonstrates  improvements  in  prediction  and 
inference  and  in  explanations  (Hernandez-Serrano  &  Jonassen,  2003). 

It  will  be  necessary  to  develop  a  repository  in  which  cases  are  scaled  for  difficulty  relative  to  the 
different  levels  of  proficiency.  For  each  level  we  will  need  cases  that  are  fairly  ordinary  at  that 
proficiency  levels,  but  also  as  "level  stretchers" —  cases  that  push  capability  beyond  the  current 
stage.  The  development  of  cases  would  permit  the  specification  of  the  features  that  make  a  case 
tough  (and  allow  investigation  of  how  they  line  up  with  the  dimensions  of  difficulty)  (Feltovich 
et  ah,  1997).  Cases  would  be  especially  evaluated  in  terms  of  “desirable  difficulties,”  to  provide 
insight  into  why  concepts  are  difficult,  which  inform  development  and  analysis  for  other  studies. 

A  corpus  of  validated  test  cases  can  be  utilized  to  gauge  the  stage  of  development  of  both 
trainees  and  practitioners.  The  Simplifying  Conditions  Method  (Reigeluth,  1999)  would  aid  in 
predicting  difficulty  and  organizing  the  results  of  rating  perceived  difficulty.  Tough  cases  will 
also  be  presented  to  apprentices,  journeymen,  and  additional  experts  for  them  to  attempt  to  solve, 
and  also  rate  for  perceived  difficulty.  In  these  tests,  participants  at  different  levels  of  proficiency 
would  be  guided  through  test  cases  in  the  same  temporal  sequence  as  the  specialist  who  helped 
create  the  case.  Information  from  the  difficult  test  case  can  be  partitioned  into  segments  in  the 
chronological  order  in  which  that  expert  originally  encountered  them.  These  segments  can  be 
presented  one  by  one  to  the  participant,  who  at  each  point  is  required  to  “think  aloud”  his  or  her 
responses  and  interpretations  (see  an  application  of  this  procedure  in  Feltovich,  Johnson,  Moller 
&  Swanson,  1984).  The  responses  can  be  compared  with  those  of  the  specialist  in  his  or  her 
original  encounter  with  the  case. 
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Cases,  once  completed  and  scaled,  would  consist  of  the  expert's  narrative,  which  would  weave 
together  the  hidden  chain  of  causally  related  events  leading  to  a  particular  problem.,  and 
associated  timelines,  analyses  of  information  requirements,  analyses  of  decision  requirements, 
and  other  resources  spanning  the  range  of  available  media  (maps,  videos,  etc.).  Cases  would  be 
scaled  for  levels  of  proficiency,  relative  to  individuals'  levels  of  ability,  both  for  routine  practice 
and  for  the  stretching  of  skill.  The  cases  would  then  be  used  to  form  instructional  materials  that 
could  range  from  paper- and-pencil  study  materials,  to  desktop  exercises,  to  full,  simulated 
scenario  experiences. 

Tough  and  rare  cases  can  also  be  used  in  unlearning,  to  reveal  knowledge  shields  (Feltovich,  et 
ah,  1997).  Rare  cases  would  be  juxtaposed  with  some  routine  or  typical  cases  in  such  a  way  that 
the  learner  could  experience  the  differences  that  produce  difficulties.  Another  application 
involves  the  fact  that  the  sociotechnical  workplace  is  constantly  changing  (Hoffman  &  Elm, 
2006).  New  technologies,  and  the  work  methods  that  they  shape,  typically  have  to  be  integrated 
with  legacy  work  methods  and  technologies.  Conceptually,  this  can  be  viewed  as  a  set  of 
scenarios  for  practice  at  adaptation,  not  merely  to  handle  familiar  and  routine  cases,  but  to 
"stretch"  the  new  technologies  and  work  methods  through  application  in  tough  cases. 

Case  experience  is  so  important  to  the  achievement  of  proficiency  that  it  can  be  assumed  that 
organizations  would  need  very  large  case  repositories  for  use  in  training  (and  also  to  preserve 
organizational  memory).  Instruction  using  cases  is  greatly  enhanced  when  "just  the  right  case"  or 
set  of  cases  can  be  engaged  at  a  prime  learning  moment  for  a  learner  (Kolodner,  1993).  This  also 
argues  for  a  need  for  large  numbers  of  cases,  to  cover  many  contingencies.  Creating  and 
maintaining  case  libraries  is  a  matter  of  organization  among  cases,  good  retrieval  schemes,  and 
smart  indexing — all  so  that  “lessons  learned”  do  not  become  “lessons  forgotten.” 


5.5  Training  Methods  and  Strategies 

For  training  across  the  proficiency  continuum,  i.e.,  going  from  Advanced  Apprentice  to 
Journeyman  levels,  it  will  be  important  to  determine  how  and  when  to  shift  from  (or  alternate 
between)  instruction  that  involves  compartmentalization/routinization  and  instruction  that 
focuses  on  interactions,  complexity  and  tough  cases.  For  training  at  Advanced  Apprentice  and 
Journeyman  levels,  it  will  be  important  to  study  issues  of  mentoring,  such  as  the  development  of 
methods  to  identify  good  mentors  and  elicit  their  knowledge  and  strategies.  In  addition,  it  will  be 
important  to  explore  alternative  forms  of  mentoring,  other  than  the  traditional  form,  such  as  the 
use  of  virtual  mentors  or  other  methods  for  accomplishing  what  mentoring  accomplishes. 


5.5. 1  Training  As  A  Function  Of  Task/Domain 

The  material  reviewed  in  this  Report  provides  some  insight  into  how  expertise  can  be  rapidly 
developed.  While  the  examples  discussed  in  these  articles  are  relatively  applied,  the  theories 
underlying  their  effects  are  general  to  the  science  of  training.  One  of  the  few  viable 
generalizations  it  is  that  the  development  of  expertise  is  contingent  upon  training  developed  for  a 
specific  context.  For  example,  the  techniques  used  to  train  skills  and  knowledge  such  as  situation 
awareness  or  team  coordination  were  developed  based  upon  specific  task  contexts  and 
constraints  (e.g.,  Endsley  &  Robertson,  2000).  The  use  of  a  networked  simulation  for  military 
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training  is  especially  important  because  it  allows  trainees  to  function  in  an  environment  that 
mirrors  the  real-world  state:  a  group-oriented  battlespace  that  requires  coordination  between 
physically  distant  teams  (Fletcher,  1999;  Cooke,  et  ah,  2007).  Similarly,  the  development  of 
teamwork  expertise  in  a  UAV  task  is  hastened  by  the  introduction  of  waylays  that  would  be 
encountered  in  the  real  world.  This  is  effectively  similar  to  the  argument  for  cognitive  fidelity, 
which  suggests  that  training  is  effective  when  it  involves  the  same  cognitive  elements  as  the  real- 
world  task  for  which  the  training  is  being  conducted. 

However,  there  is  more  to  these  studies  than  a  simple  recommendation  of  cognitive  fidelity.  The 
Cooke,  et  al.  (2007)  results  reveal  that  team  training  for  expertise  is  best  conducted  with 
perturbations,  even  when  the  real-world  task  would  seldom  encounter  such  perturbations.  This 
allows  training  to  target  elements  of  a  task  that  rarely  occur,  but  are  critical  to  understand  and 
solve  when  they  do  arise.  In  the  case  of  piloting,  this  sort  of  expertise  can  resolve  situations  that 
would  otherwise  result  in  catastrophe. 

Thus,  the  development  of  expertise  requires  not  only  a  training  intervention  that  mirrors  the 
cognitive  tasks  inherent  to  the  real-world  task,  but  also  the  development  of  skills  that  are  rarely 
used  but  critical  to  success.  The  question  that  remains,  then,  is  how  to  balance  these  two 
seemingly  divergent  needs.  It  is  possible  to  envision  a  system  that  combines  the  various 
techniques  described  in  the  reviewed  articles.  For  example,  a  networked  team  simulation  that 
introduces  perturbations  (whether  through  alterations  in  team  makeup  or  the  addition  of  task¬ 
relevant  hindrances)  as  part  of  a  comprehensive  training  course  would  create  a  useful  training 
tool  for  a  group  context. 

Another  domain  factor  that  plays  into  training  is  feedback.  Studies  of  diverse  domains  of 
expertise  reveal  a  great  variability  between  domains  (and  specializations  within  domains)  in  the 
extent  to  which  workers  receive  any  feedback,  let  alone  timely,  high-quality  feedback  (e.g., 
Salas,  Nichols,  &  Driskell,  2007).  In  part  this  may  explain  why  it  can  be  difficult,  in  some 
domains,  to  identify  what  counts  as  expertise.  The  inherent  nature  of  the  domain  can  make  it 
impossible  for  the  practitioner  to  receive  timely  feedback  (e.g.,  long-term  weather  forecasting, 
intelligence  analysis  for  policy  projection).  This  may  be  one  reason  why  it  can  take  a  decade  or 
more  for  an  individual  to  achieve  expertise. 

For  some  domains  of  study,  it  may  be  appropriate  to  review  debriefing  methods  to  determine 
ways  of  shaping  the  debriefings  so  as  to  contribute  to  the  acceleration  of  proficiency.  Klein 
(2010)  has  suggested  a  notion  of  “deliberate  performance”  as  a  counterpart  to  the  notion  of 
deliberate  practice.  A  Deliberate  Performance  method  would  collect  and  use  feedback  during  and 
immediately  after  actual  work  experiences.  Deliberate  performance  attempts  to  turn  people  into 
reflective  practitioners.  Deliberate  performance  analysis  would  have  the  goal  of  encouraging  the 
learner  to  become  a  reflective  practitioner  through  activities  including  explanation  and 
extrapolation. 

5.5.2  Training  as  a  Function  of  Proficiency  Levei 

We  have  a  strong  foundation  of  knowledge  and  theory  on  expertise.  For  example,  in  the  context 
of  improving  learning  in  classroom  settings,  Lajoie  (2003)  suggested  that  “identifying  what 
experts  know  can  help  determine  the  trajectory  towards  competence  for  that  task”  (p.  21)  and 
guide  the  development  of  content  appropriate  for  differing  stages  of  learning.  However,  there  is  a 
considerable  qualitative  and  quantitative  gap  in  between  the  knowledge  and  skills  of  the  expert 
and  those  of  the  novice.  For  accelerated  learning  what  is  critical  are: 
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1.  The  gap  from  initiate  (individual  receiving  introductory  training)  to  senior  apprentice, 

2.  The  gap  from  apprentice  to  journeyman,  and 

3.  The  gap  from  journeyman  to  expert. 

Fiore,  Hoffman  and  Salas  (2008)  presented  the  following  set  of  testable  hypotheses  concerning 
training  as  suited  to  novices,  initiates,  apprentices  and  journeyman.  For  the  focus  of  the  present 
Report,  we  note  their  claim  that  at  the  apprentice  level  mentoring  is  especially  important,  where 
appropriate  actions  and  decisions  are  illustrated  in  the  context  of  the  actual  operational 
environment  and,  when  warranted,  the  learner  is  guided  in  his  own  performance  (Collins,  Brown, 
&  Newman,  1989).  Training  needs  to  reveal  and  resolve  the  “knowledge  shields,”  which  are 
strategies  or  rationalizations  for  sticking  with  simplified  understandings  even  when  evidence 
suggests  that  concepts  are  wrongly  understood  (Feltovich,  et  ah,  1997).  Here  the  learner  must 
recognize  incorrect  understanding  or  assumptions  and  resolve  any  misconceptions  he  may  hold. 
As  such,  corrective  feedback  via  expert  mentoring  or  within  a  simulation-based  learning 
environment  is  critical. 

At  the  Advanced  Apprentice  and  Journeyman  levels,  notions  of  CFT  and  CTT  come  into  play, 
related  to  their  emphasis  on  learning  at  advanced  levels.  The  learner  needs  to  be  presented  with 
environments  presenting  highly  variable  sample  tasks  and  rare,  difficult  examples  that  challenge 
the  learner’s  concepts  and  categories.  Adapting  to  such  challenges  leads  to  a  richer 
representation  of  a  domain  and  supports  flexibility  in  later  problem-solving  situations.  Given 
their  developing  autonomy,  those  at  the  Advanced  Apprentice  and  Journeymen  levels  might 
independently  use  simulation-based  environments  to  train  on  these  types  of  learning  situations. 

It  has  been  said  that  apprentices  make  the  same  mistake  twice,  journeyman  make  the  same 
mistake  once  and  experts  work  until  they  never  make  mistakes.  But  domain  specialists  of  high 
proficiency  are  intrinsically  motivated  so  seek  out  corrective  feedback  that  allows  them  to 
perceive  their  errors  (Sonnentag,  2000).  The  more  highly  experienced  and  motivated  domain 
practitioners  seek  corrective  feedback  from  coworkers.  When  everything  works  the  way  it  is 
supposed  to,  one  is  less  likely  to  receive  feedback  about  what  did  not  work  or  what  might  have 
been  done  better.  As  such,  experts  seek  out  feedback  pointing  out  targets  for  improvement. 

What  counts  as  useful  feedback  depends  on  the  proficiency  level  of  the  individual  whose 
performance  is  being  evaluated  or  who  is  in  training.  It  has  been  speculated  that  novices  learn 
more  from  successes  because  they  need  help  figuring  out  what  to  do,  whereas  experienced 
people  learn  more  from  failures  because  they  have  already  mastered  the  routines  and  are  in  a 
position  to  make  use  of  the  information  found  in  errors.  In  fact,  would-be  experts,  when  they 
practice,  will  inevitably  encounter  more  failures  than  lesser  learners  because  they  focus  on 
developing  skills  beyond  those  with  which  they  are  already  comfortable. 

CFT  and  CTT  suggest  feedback  should  help  the  learner  transcend  the  inclination  to  invoke  a 
knowledge  shield  (i.e.,  rationalize  away  a  misunderstanding),  and  un-leam  concepts  or  notions 
that  incorrectly  simplify  their  understanding  of  the  domain. 

Research  shows  that  for  promoting  the  achievement  of  proficiency,  feedback  needs  to  be: 

•  Timely,  that  is,  close  enough  in  time  to  the  point  of  decision  or  action  to  be  informative 
but  not  so  close  as  to  prevent  the  decision  maker  from  thinking  through  what  had  just 
happened. 
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•  Satisfying  to  Sensemaking  and  Plausibility  Judgment,  in  terms  of  causal  explanation, 

•  Informative  of  Outcome  and  Process,  explaining  what  happened  and  analyzing  the 
decision  maker’s  reasoning. 

5.5.3  Scenario-Based  Training 

In  section  5.4  above  about  Knowledge  Management,  we  discussed  the  need  to  develop  libraries 
of  cases,  scaled  for  difficulty,  and  that  such  a  corpus  could  be  used  as  training  material.  People 
learn  what  they  practice;  hence,  to  achieve  adaptive  expertise,  they  should  practice  in  many 
ways,  across  many  kinds  of  cases  (including  ones  that  involve  the  same  basic  principles,  but  in 
different  kinds  of  contexts),  using  many  kinds  of  conceptual  tools,  points  of  view,  mental 
organizational  structures,  investigation  and  practice  strategies,  and  the  like  (e.g.,  Feltovich,  et  ah, 
1997).  The  modes  and  means  of  training  should  engage  real  work  practice,  the  challenges, 
contexts,  and  duties  of  the  job,  to  the  greatest  extent  possible. 

Training  for  the  acceleration  of  proficiency  in  domains  of  complex  cognitive  work  will  have  to 
include  an  appreciation  of  the  importance  of  causal  reasoning,  anticipatory  thinking,  and  problem 
detection.  In  order  for  the  practitioner  to  be  better  able  to  handle  complexity,  especially 
unexpected  complexity,  training  must  involve  helping  people  acquire  a  more  powerful  toolbox, 
and  acquire  skill  at  knowing  how  to  build  entirely  new  tools  when  tough  cases  arise. 

Given  that  experts  often  extrapolate  from  patterns  of  cues  and  think  in  terms  of  rich  causal 
models,  the  simulation  scenarios  would  be  set  up  such  that  the  learner  is  sometimes  led  and 
sometimes  misled  in  seeking  solutions  (of  course,  while  not  deviating  from  authenticity).  In  the 
case  of  being  misled,  the  idea  is  that  the  simulation  purposely  misguides  a  learner  into  a  common 
solution,  but  the  reality  is  that  the  error  is  due  to  a  little  known  or  subtle  issue.  For  example,  with 
regard  to  ordering,  an  appropriately  crafted  scenario  can  manage  temporal  sequencing  to  support 
problem  solving.  This  could  juxtapose  real-world  time  with  narrative  time,  so  that  the  learner  can 
experience  the  event  through  the  perspective  of  the  expert  (real-world  time),  and  through  the 
more  omniscient,  third  person  perspective  of  narrative  time,  that  is,  presenting  events  in  alternate 
directions  so  as  to  highlight  or  prominently  position  certain  occurrences  for  interpretation  by  the 
learner.  Simulations  allow  for  tremendous  flexibility  in  manipulations  of  fidelity — differing 
kinds  and  amounts  of  contextual  richness.  These  variations  are  important  because  they  can 
emphasize  particular  elements  of  the  scenario  to  the  learner. 

5.5.4  Simuiation-Based  Training 

Throughout  this  Report  we  have  referred  to  training  using  technology  spanning  games,  to 
computer  simulations,  to  virtual  reality.  Characteristics  of  effective  scenario-based  training  are 
listed  in  Table  4.8,  above,  and  include  realistic  environments,  mentoring,  and  focused  feedback. 
A  clear  consensus  is  that  simulation-based  training  is  critical,  necessary,  and  has  at  yet  untapped 
potential  (Fletcher,  2009). 

Simulation-based  training  is  an  instance  of  minimizing  the  transfer  distance. 

The  expert  knows  there  is  no  substitute  for  going  out  to  look  at  the 
problem  yourself...  out  there  smelling  the  ashes.  [We  need  to] 
bottle  these  experiences  to  simulate  them...  [the]  look  and  feel  and 
smell  of  critical  events  is  what  defines  the  problem  (Ziebell,  2010). 

The  modes  and  means  of  training  should  engage  realistic  work  practice,  including  the  challenges. 
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contexts,  and  duties  of  the  job.  Sometimes  less  realistic  practice  is  effective,  as  well  as  more 
cost-effective,  especially  for  earlier  stages  of  training.  This  might  also  be  true  for  advanced 
learning,  in  certain  circumstances.  But  to  generalize,  training  at  advanced  levels  should  be  a 
short  “transfer  distance”  from  the  work  context.  This  encompasses  everything  from  instructional 
settings,  tools,  workshop  exercises,  problem-based  learning,  and  real-time  mentor  guidance,  as  in 
apprenticeship  on  the  job  (Collins,  Brown,  &  Newman,  1989;  Brown,  Collins,  &  Duguid,  1991). 
Learners  can  see  the  value  of  what  they  are  trying  to  learn  because  that  value  is  apparent  in  the 
work  they  are  actually  doing,  and  the  transfer  (of  knowledge)  conditions  and  application  criteria 
are  likewise  relatively  salient  (e.g.,  Bransford,  et  ah,  1990;  Coulson  &  Feltovich,  1993;  Lesgold 
&  Nahemow,  2005). 

Scenario  Content  -  What  is  the  problem  being  represented?  First,  with  regard  to  scenario 
content,  the  scenarios  would  need  to  address  integrative  complexity  arising  from  the  interaction 
of  legacy  and  modern  systems.  Conceptually,  this  can  be  viewed  as  a  set  of  scenarios  where  the 
troubleshooting  and  diagnosis  sensemaking  systems  with  new  systems,  or,  more  likely,  new  with 
old  systems.  Further,  these  rare  cases  would  be  juxtaposed  with  typical  cases  in  such  a  way  that 
the  learner  can  experience  the  differences  that  produced  the  problem  and  recognize  the 
differences  of  the  diagnostic  cues  in  each. 

The  simulation  scenarios  would  be  set  up  such  that  the  learner  is  sometimes  lead  and  sometimes 
misled  in  their  solution  seeking.  In  the  case  of  being  misled,  the  idea  is  that  the  simulation 
purposely  misguides  them  into  a  common  solution  but  the  reality  is  that  the  error  is  due  to  a  little 
known  issue  associated  with  an  interaction  between  a  legacy  system  and  modern  devices.  This 
will  address  the  problem  demonstrated  by  Besnard  and  Bastien-Toniazzo  (1999)  who  showed 
that  experts  sometimes  made  errors  in  electronics  troubleshooting  by  developing  a  fixation  on  a 
particular  component  that  suggested  it  was  at  fault.  Training  needs  to  encourage  “systems 
thinking”  (Linou  &  Kontogiannis,  2004)  that  trains  about  functional,  temporal,  and  causal-link 
information.  Patrick  (1993)  found  that  systemic  information  supported  both  retention  and  the 
ability  to  reconstruct  the  task  cues. 

Narrative  Structure  -  How  does  the  story  get  told?  Given  the  multi-faceted  nature  of  telling 
stories  and  the  complexity  of  the  operations  and  maintenance  tasks,  research  needs  to  attend  to 
the  manner  in  which  events  and  episodes  are  woven  together  within  the  simulation.  Narrative,  as 
the  formal  apparatus  of  storytelling,  should  be  used  to  weave  together  the  sometimes  hidden 
complexities  of  causally  inter-related  events  leading  to  a  particular  problem.  More  formally, 
narrative  determines  how  the  story  gets  told,  that  is,  who  tells  the  story,  the  degree  of 
omniscience  held,  the  ordering  of  the  events,  and  descriptiveness  of  the  setting.  For  example, 
with  regard  to  the  ordering,  an  appropriately  crafted  scenario  can  manage  temporal  sequencing  to 
support  problem  solving.  This  could  juxtapose  real-world  time  with  narrative  time  so  that  the 
learner  can  experience  the  event  through  the  perspective  of  the  expert  (real-world  time),  and 
through  the  more  omniscient,  3  person  perspective  of  narrative  time,  that  is,  presenting  events 
in  alternate  directions  so  as  to  highlight  or  prominently  position  certain  occurrences  for 
interpretation  by  the  learner. 

Story  Representation  -  How  is  the  story  represented?  Simulations  allow  for  tremendous 
flexibility  in  manipulations  of  fidelity,  which  can  provide  differing  amounts  of  contextual 
richness.  These  variations  are  important  because  they  can  differentially  emphasize  particular 
elements  of  the  scenario  to  the  learner. 
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5.5.5  Training  for  Probiem  Recognition 

Much  expert  problem  solving  is  done  by  recognition  of  problem  types  and  difficulty  (e.g. 
Feltovich  et  al.,  1997;  Hatano  et  al.,  2003;  Klein  &  Baxter,  1998).  Experts'  categories  for  fall 
also  at  a  functional  rather  than  literal  surface  feature  level  (Glaser,  1987).  Thus,  experts  can 
rapidly  evaluate  a  situation  and  determine  an  appropriate  plan  of  action,  a  phenomenon  called 
“recognition-primed  decision  making”  (Klein,  1993,  1998).  Within  the  first  second  of  exposure 
to  a  novel  chess  position,  chess  experts  can  extract  important  information  about  the  relations  of 
the  chess  pieces’  positions,  and  begin  identifying  promising  moves  (Charness,  et  al.,  2001). 

Yet  experts  also  realize  when  their  routine  ways  are  inadequate  to  the  task.  Typically  this  switch 
requires  notice  that  something  is  unusual  about  the  problem.  By  using  cases  that  represent  the 
typical,  along  with  tough  cases,  training  can  be  in  the  detection  of  unusualness  (e.g.,  switching 
from  mainly  recognition-primed  decision  making  to  some  more  deliberative  form  of  reasoning), 
since,  almost  by  definition,  tough  cases  will  contain  tricky  or  unusual  aspects.  Training  can 
manipulate  the  operative  cues  within  the  cases  that  indicate  the  cases  are  unusual — from  low 
level  cues  that  are  consistent  with  simple  interpretations  (the  “easy”  side  of  the  dimensions  of 
difficulty,  static  compartmentalized,  etc.)  to  ones  in  which  the  operative  cues  are  complex  (e.g., 
relational,  dynamic,  interdependency-based). 

It  is  also  clear  that  these  cognitive -perceptual  abilities  take  time  to  develop  and  result  in  new 
(and  effective)  ways  in  which  experts  view  and  interpret  the  world  (Chi,  Feltovich,  &  Glaser, 
1981;  Klein  &  Hoffman  (1993).  Since  this  area  of  development  is  so  crucial  to  the  development 
of  expert  performance,  it  must  also  be  a  focus  of  training  for  skill  improvement  (e.g.,  Fadde, 
2007),  and  it,  like  other  skills  discussed,  is  probably  most  amenable  to  case-based  kinds  of 
leaming/practice . 

5.5.6  Training  for  Perceptuai  Learning 

Skill  at  the  perception  and  integration  of  data  is  critical  for  many,  if  not  most  domains  of 
complex  cognitive  work.  Experts  can  perceive  things  that  are  invisible  to  novices  (Klein  & 
Hoffman,  1992).  While  it  has  proven  difficult  to  capture  “perceptual  learning”  in  the  lab  is  a 
challenge,  the  results  of  perceptual  learning  have  been  shown  in  studies  of  expertise  in  a  variety 
of  domains  ranging  from  neonatal  critical  care  nursing  to  commercial  fishing.  Studies  have 
looked  at  domains  where  perceptual  skill  is  paramount,  such  as  radiology  (see  Goldstone,  1998; 
Fesgold,  et  al.,  1988)  and  baggage  screening  (Fiore,  et  al.,  2000).  We  know  that  experts’ 
knowledge  organization  involves  finer  gradations  of  functional  categories,  that  is,  “basic  object- 
level”  categories  fall  at  a  finer  level  than  for  non-experts.  For  example,  limestone  is  simply  a 
kind  of  rock  to  most  people,  but  to  the  expert,  there  are  many  variants  that  inform  of  geological 
dynamics  (e.g.,  tilted  thinly  interbedded  limestone-shale  with  limestone  predominating). 

Perceptual  learning  is  not  just  about  the  perception  of  cues  or  the  reckoning  of  variables,  it  is 
about  their  meaningful  integration.  A  case  study  from  the  U.S.  Army  Corps  of  Engineers  is 
presented  in  Textbox  5.1. 
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Textbox  5.1  A  case  study  in  perceptual  learning. 


Terrain  analysis  is  a  process  in  which  aerial  photographs  are  interpreted  to  assess  soils, 
bedrock,  vegetation  patterns,  drainage  patterns,  and  the  like.  In  one  experiment  (Hoffman, 
1987),  an  expert  was  presented  a  photograph  for  inspection,  but  was  only  allowed  two 
minutes  of  viewing  time.  Ordinarily,  the  full  systematic  terrain  analysis  process  can  take 
hours.  After  the  viewing  period,  the  expert  was  allowed  two  minutes  in  which  to  report 
everything  that  could  be  remembered  about  the  photo.  In  one  particular  trial,  the  expert 
began  his  retrospection  by  asserting  that  any  personnel  sent  to  the  depicted  area  would  need 
to  be  prepared  for  certain  types  of  bacterial  infection.  The  experimenter's  response  was,  "You 
can  see  bacteria  in  a  pond  taken  from  40,000  feet?"  Here  was  the  reasoning  sequence  that 
the  expert  recounted:  The  photo  covered  an  area  of  tropical  climate.  The  vegetation  was 
mature  and  uniform,  so  the  contours  to  the  top  of  the  underlying  soil,  and  since  the  soil  layer 
would  be  relatively  thin,  the  contour  to  the  tree  canopy  reflected  the  underlying  bedrock, 
which  appeared  to  be  tilted  interbedded  limestone.  The  bedrock  also  determined  the  pattern 
to  the  streams  and  ponds,  and  there  appeared  one  pond  that  did  not  have  a  tributary  running 
away  from  it.  Given  the  climate,  the  vegetation  (tropical  legumes)  and  the  stagnant  water,  the 
presence  of  bacteria  was  a  virtual  certainty.  This  appears  to  be  a  long  chain  of  inferences, 
dependent  on  a  great  deal  of  declarative  knowledge.  Yet,  in  the  actual  experimental  trial,  the 
expert's  judgment  was  the  sort  that  one  might  be  inclined  to  call  "direct"  or  "immediate,"  a 
perceptual  phenomena  rather  than  a  linear,  deliberative  process. 


The  informative  cues  might  be  separable  in  the  sense  that  they  can  be  operationally  defined  and 
measured  independently  of  one  another,  but  they  link  together  for  principled  reasons  and  relate 
to  one  another  in  meaningful  ways.  In  functional  terms,  they  are  integral  (Phelps  &  Shanteau, 
1978).  Goldstone  (1998)  has  referred  to  this  as  a  process  of  “unitization,”  in  which  a  “single 
constructed  unit  represents  a  complex  configuration”  (p.  585). 

Often,  the  patterns  that  bear  meaning  cannot  be  defined  in  terms  of  the  simple  presence  or 
absence  (or  the  values)  of  each  of  a  set  of  separable  cues  (which  is  how  “pattern  recognition  was 
traditionally  defined).  Meaningful  patterns  are  sometimes  defined  by  the  relations  among 
functionally  integral  cues.  Ludwig  Wittgenstein  was  getting  at  this  with  his  notion  of  “featureless 
family  resemblances.”  For  instance,  the  expert  weather  forecaster  can  see  a  “gate-to-gate 
signature”  in  radar  that  is  a  clue  to  tornado  formation.  This  signature  is  a  function  of  a  difference 
(is  a  relation)  in  relative  velocity  (is  a  relation)  of  proximal  (is  a  relation)  winds,  with  strong  (is  a 
relation)  winds  moving  toward  the  radar  (is  another  relation)  in  very  close  proximity  (is  yet 
another  relation)  to  strong  (is  a  relation)  winds  that  are  moving  away  from  (and,  again,  another 
relation)  the  radar.  Clearly,  a  considerable  nexus  of  relations.  The  patterns  that  are  meaningful  to 
experts: 

•  Sometimes  involve  individual  cues, 

•  Sometimes  involve  sets  of  separable  cues  with  some  cues  being  necessary  and  some 
being  sufficient, 

•  Sometimes  involve  patterns  that  can  be  defined  in  terms  of  combinations  of  cues, 

•  Sometimes  involve  patterns  defined  in  terms  of  relations  among  cues, 

•  Sometimes  are  featureless  family  resemblances  where  cues  are  neither  necessary  or 
sufficient  when  considered  individually,  and 
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•  Sometimes  involve  meaning  that  resides  in  the  relations  among  cues  that  are  integral,  or 
cue  configurations. 

In  complex  sociotechnical  work  systems,  decision  makers  rely  on  multiple  data  types,  and  data 
that  are  also  in  continuous  flux.  This  takes  the  traditional  notion  of  “pattern  recognition”  to 
entirely  new  levels,  because  the  meaningful  patterns  are  typically  dynamic,  that  is  the  important 
meaning  and  relational  information  exists  only  over  time. 

In  aerial  photo  interpretation,  the  patterns  that  experts  perceive,  even  in  static  images,  are 
dynamic  (Hoffman  &  Pike,  1995).  Experts  perceive  processes.  Furthermore,  the  dynamic  cue 
configurations  sometimes  exist  only  across  multiple  data  types.  The  patterns  experts  perceive 
sometimes  do  not  exist  in  individual  data  types.  Indeed,  the  really  critical  information  often  is 
"trans-modal" — exists  only  across  data  types.  For  instance,  in  weather  forecasting  the  radar 
images  are  not  the  only  thing  that  is  guiding  sensemaking  activity  and  shaping  the  forecaster’s 
formation  of  a  mental  model.  A  great  many  other  data  types  are  involved,  such  as  satellite 
images,  computer  model  outputs,  wind  fields,  pressure  data,  and  so  on.  The  “Aha!”  moment 
might  come  when  viewing  radar,  but  the  mind  had  been  prepared  in  the  sense  that  a  mental 
model  was  based  on  an  integration  of  meaningful  patterns  that  only  exist  across  data  types. 
Fikewise,  the  expert  terrain  analyst  makes  determinations  when  viewing  aerial  photos,  but  has 
engaged  in  systematic  analysis  of  other  data,  such  as  maps. 

To  raise  the  benchmark  for  proficiency  even  higher,  one  must  take  into  account  the  “Moving 
Target  Rule”  (Balias,  2007;  Hoffman  &  Woods,  2005):  The  sociotechnical  workplace  is 
constantly  changing,  and  change  the  technologies  in  the  workplace  will  entail  change  in  the  work 
to  be  accomplished,  even  if  the  domain  constraints  remain  constant.  Change  in  cognitive  work 
comes  because  of  changes  in  goals  (i.e.,  new  tasks,  new  challenges)  but  especially  because  of 
changes  in  technology,  including  changes  in  data  types  and  display  types.  For  instance,  the 
NEXRAD  radar  has  revolutionized  radar  meteorology  and  forecasting,  and  new  radar  algorithms 
are  being  introduced  all  the  time,  resulting  in  new  data  products  (hence,  new  displays),  and  new 
combinations  of  data  types  (e.g.,  “Sat-Rad”  displays  that  combine  satellite  imagery  with  radar 
data). 

In  the  modern  sociotechnical  work  context,  the  expert  must  engage  in  frequent,  if  not  nearly 
continuous  perceptual  re-learning.  Patterns  previously  learned  and  perceived  in  one  way  come 
to  be  perceived  in  a  new  way.  Experts  engage  in  perceptual  re-learning  of  dynamic  information 
defined  over  sets  of  integral  cues  that  are  trans-modal  (they  exist  over  different  data  types). 

A  critical  scientific  gap  is  the  paucity  of  research  attempting  to  capture  the  perceptual  re-learning 
of  dynamic  cue  configurations  that  exist  across  multiple  data  types.  We  need  to  know  more  about 
how  it  happens  in  domains  of  expertise.  Available  research  gives  only  the  barest  of  clues. 

Any  method  for  accelerating  the  achievement  of  proficiency  could  consider  supporting  the 
processes  of  perceptual  re-leaming.  In  domains  of  expertise  where  perceptual  skill  is  paramount, 
such  as  baggage  screening  or  weather  forecasting  (Fiore,  et  ah,  2004),  it  seems  reasonable  to 
speculate  that  providing  critical  exemplars  of  targets  makes  the  perceptual  learning  process 
possible,  but  it  may  not  accelerate  it.  One  might  expect  that  a  requirement  to  engage  in 
perceptual  re-learning  will  be  effortful  and  disruptive  even  for  experienced  domain  practitioners. 
But  experienced  practitioners  should  be  able  to  re-acquire  expertise  in  less  time  than  less- 
experienced  practitioners  (students,  journeymen)  acquire  it. 
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5.5.7  Training  for  Robustness,  Resi Hence  and  Adaptivity 

In  domains  of  dynamics  and  complexity,  the  work  (including  the  technologies)  is  continually 
changing  and  thus  mental  models  have  to  change  (Mumaw,  et  ah,  2000).  Robustness  is  the 
ability  to  maintain  effectiveness  across  a  range  of  tasks,  situations,  and  conditions.  Resilience  is 
the  ability  to  recover  from  a  destabilizing  perturbation  in  the  work  as  it  attempts  to  reach  its 
primary  goals.  Adaptivity  is  the  ability  to  employ  multiple  ways  to  succeed  and  the  capacity  to 
move  seamlessly  among  them.  Fundamental  to  the  achievement  of  robustness,  resilience  and 
adaptivity  is  the  opportunity  to  practice  at  problems  that  stretch  current  competency  (Ericsson  & 
Lehmann,  1996;  Feltovich,  et  ah,  1997).  Training  cannot  oversimplify.  In  trying  to  train  for  the 
ability  to  handle  complexity,  there  have  traditionally  been  two  general  approaches.  The  first  is  to 
simplify  and  gradually  add  in  complex  elements.  While  this  seems  highly  intuitive,  and  may  be 
the  most  common  mode  in  education  and  training,  for  highly  complex  material  it  can  backfire. 
Learners  can  get  stuck  in  the  simplifications  and  petrify  there  (e.g.,  Feltovich  et  ah,  1989;  2001, 
Spiro,  et  ah,  1989).  Another  is  to  address  the  real  complexity,  but  at  the  same  time  make 
available  ample  expert  and  mentoring  help.  Some  of  the  most  outstanding  real  examples  of 
experience  and  learning  acceleration  for  highly  complex  material  and  tasks  have  used  this  latter 
approach  (Bonsangue,  1993;  Lesgold,  Lajoie,  Buzno,  &  Eggan,  1992;  Palincsar,  Spiro  et  ah,  in 
press).  These  programs  have  used  practice  problems  and  materials  that  were  at  least  as  hard  as  or 
harder  than  what  the  learners  were  prepared  for. 

Too  achieve  adaptability,  training  needs  to  utilize  “concept-case  coupling.”  Training  must 
involve  the  learning  of  conceptual  knowledge,  i.e.,  regularities,  general  principles,  laws,  and  so 
forth.  These  are  the  abstractions  that  apply  across  cases  of  application  of  these  principles  and, 
thus,  are  a  vehicle  for  transfer.  But  because  these  are  abstractions,  such  conceptual  knowledge 
must  be  applied  to  the  details  of  a  case  of  application,  details  that  require  the  contextual  tailoring 
of  the  principle  for  application  in  some  cases  or  can  even  render  a  concept  not  applicable  at  all. 
Training  must  include  considerable  opportunity  for  practicing  the  application  of  conceptual 
knowledge  across  ranges  of  cases,  for  example,  through  guided  apprenticeships  or  the  use  of 
case-based  practice  technologies  (e.g.,  Collins,  Brown,  &  Newman,  1992;  Lesgold  &  Nahemow, 
2005;  Spiro,  Collins,  Thota,  &  Feltovich,  2003). 

People  learn  pretty  much  what  they  practice.  Hence,  to  achieve  proficiency  and  high  proficiency 
within  a  domain,  people  should  practice  in  many  ways,  across  many  kinds  of  cases  (including 
ones  that  involve  the  same  basic  principles,  but  in  different  kinds  of  contexts),  using  many  kinds 
of  conceptual  tools,  points  of  view,  mental  organizational  structures,  investigation  and  practice 
strategies,  and  the  like  (e.g.,  Bransford,  et  ah,  1979  Spiro  et  ah,  1989,  1992).  For  example, 
individuals  should  cross-train  with  others  in  the  organization,  those  in  different  work  positions, 
so  one  can  see  how  the  world  looks  from  that  vantage  point. 

All  these  considerations  lead  to  a  particular  simulation-based  training  method. 

The  Operational  Simulation  Method 

The  OPSIM  method  developed  by  WRTI,  Inc.  (DiBello  &  Missildine,  2008;  DiBello,  Missildine, 
&  Struttman,  2008)  was  discussed  above  in  Section  2.7. a  as  an  example  of  a  successful 
demonstration  of  acceleration,  successful  in  a  number  of  different  business  domains.  It  would  be 
prudent  to  attempt  to  adapt  this  method  to  other  domains  and  training  contexts.  The  method  is  a 
simulation  exercise  in  which  participants  engage  with  a  “reality  analogous”  version  of  their 
organizational  environment  in  a  compressed  time  period.  In  doing  so,  participants  confront 
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organizational  failures  and  rehearse  strategic  solutions  while  being  held  accountable  to 
outcomes.  Participants  work  as  a  team  to  solve  difficult  problems,  and  in  so  doing  leverage  and 
share  their  individual  expertise.  This  leads  to  novel  solutions,  such  as  new  ways  of  coordinating 
decision  making  among  multiple  organizational  entities,  and  implementing  a  new  information 
technology  or  resource  planning  system. 

Essential  to  the  success  of  method  is  reorganizing  participants’  “default  mental  models”  (or 
“business  as  usual”  approaches  to  solving  a  problem)  in  order  to  meet  the  novel  demands  of  the 
goals  that  have  been  designed  into  the  environment.  To  achieve  this,  however,  the  default  mental 
model  must  first  be  expressed  and  analyzed.  Because  cognition  and  activity  are  inextricably 
linked,  a  workplace  cannot  simply  add  new  mental  models  on  top  the  old.  Just  as  architects  must 
often  tear  down  old  structures  before  they  can  rebuild,  so  the  individual  mind  and  the 
organizational  brain  must  clear  away  the  default  mental  model,  before  novel  ways  of  working 
can  emerge.  This  certainly  fits  with  the  ideas  of  the  CFT/CTT  merger. 

In  the  training,  the  participants  must  actually  run  an  emulated  version  of  their  organization  with 
several  interacting  functions,  attempting  to  meet  or  exceed  their  goals  under  a  time  pressure  (e.g., 
often  a  “month”  is  20  min  in  a  manufacturing  exercise).  The  immersive  simulation  aspect  of  the 
exercise  is  what  activates  the  default  mental  model.  Participants  engage  “as  though”  they  are  at 
work.  Adding  to  stress,  participants  “play”  for  8  hours  or  more  at  a  stretch  while  getting 
dynamic,  real  time,  granular  feedback  about  their  performance  in  a  compressed  time  period, 
within  the  micro  environment.  At  any  moment,  participants  can  measure  their  performance 
against  a  hypothetical  ideal  in  order  to  tell  how  far  they  are  from  the  ideal  on  any  of  the 
parameters  being  measured  during  the  exercise.  Invariably,  any  major  organizational  breakdown 
that  is  present  in  the  real  life  work  environment  is  replicated  by  the  participants  during  the 
exercise;  i.e.,  the  group  invariably  “fails”  in  a  way  that  mirrors  their  current  problems  in  the  real 
workplace. 

Of  course,  this  sounds  remarkably  similar  in  overall  structure  to  the  way  exercises  are  conducted 
at  the  National  Training  Center. 

In  the  Day  2  of  the  exercise,  participants  are  given  feedback  based  on  various  performance 
measures  from  the  activities  of  the  previous  day.  The  participants  discuss:  1)  what  went  wrong, 
2)  what  we  will  do  differently  this  time,  and  3)  how  will  we  know  it  is  working.  They  then  run 
through  the  exercise  a  second  time.  They  must  make  their  “new”  plan  work  under  the  same  time 
pressure  as  before.  Invariably,  participants  doing  the  exercise  a  second  time  after  failing  once 
manage  to  meet  or  exceed  the  goals  and  develop  innovative  and  detailed  solutions  that  prevent 
the  problems  encountered  before. 

WRTI  has  found  that  the  gains  are  transferred  back  to  real  work  as  fully  rehearsed  tactical  plans 
and  results  are  normally  seen  in  the  real  work  places  within  weeks  or  months. 

Scenarios  such  as  war  games  and  flight  simulators  often  do  an  amazing  job  at  replicating  real- 
world  environments.  However,  these  kinds  of  simulations  are  primarily  useful  for  learning  a 
specific,  standardized  set  of  procedures,  plans,  or  certain  emerging  strategies.  These  kinds  of 
simulations  often  pre-designed  a  particular  method  to  which  the  participants  must  adapt.  The 
purpose  of  these  simulations  is  not  to  allow  new  design  solutions  to  emerge.  The  OpSim^M 
method  has  several  features  that  set  it  apart  from  other  simulation  exercises. 
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•  It  is  suited  to  training  for  work  in  complex  sociotechnical  systems  where  adaptivity  and 
resilience  are  critical. 

•  It  is  suited  for  complex  coordination  and  implementation  scenarios  in  which  people  have 
to  solve  multiple  problems  at  once. 

•  It  is  dynamic  enough  to  synthesize  solutions  for  both  high  level,  strategic  problems  and 
front-line,  operational  issues  rooted  in  entrenched,  default  mental  models. 

These  ideas  and  findings  hearken  to  those  of  Schmidt  and  Bjork  (1992;  see  also  Valkeavaara, 
1999),  that  schemes  of  learning  which  are  in  the  short  run  harder,  and  often  even  less  effective  in 
the  short  run,  can  yield  long  term  advantages  in  both  retention  and  transfer  'punch.' 

Decision  Making  Exercise 

The  Decision  Making  Exercise  (DMX)  method  was  developed  initially  to  train  high-level 
decision  making  skills  for  military  commanders  (Klein,  2003).  A  similar  method  was  created  by 
Lia  DiBello  (2006).  The  trainer  provides  information  about  a  critical  event,  asks  the  candidate  to 
make  a  prediction  about  some  unifying  variable,  shows  the  actual  data,  and  then  continues 
requesting  predictions  and  providing  feedback  as  the  scenario  unfolds.  The  issue  is  not  whether 
the  person  can  generate  an  accurate  prediction  from  the  start.  Rather,  the  issue  is  how  quickly  the 
person  improves.  The  best  people  will  revise  their  mental  models  to  improve  their  predictive 
accuracy  based  on  what  they  can  learn.  In  the  versions  of  the  method  conducted  by  Klein 
Associates,  trainees  in  military  command  were  presented  scenarios  that  consisted  of  a  map  or 
diagram  plus  a  one-page  description  of  events  thus  far,  unexpected  continuations,  and  the 
challenge  to  forge  a  decision  on  the  spot.  These  exercises  are  intended  to  be  played  in  groups, 
with  a  facilitator,  but  they  can  also  be  played  on  an  individual  basis.  As  the  participants  walk 
through  the  re-creation  of  a  tough-case  decision  situation,  they  can  be  fed  information  as  needed, 
or  as  it  would  unfold  in  actual  situations  (but  time  compressed).  Note  that  these  exercises  differ 
from  decision  making  games  in  that  there  is  no  one  single  best  or  good  solution.  Rather,  the 
participants  must  navigate  through  trade-offs. 

While  not  having  single  "best"  answers,  exercises  can  be  formed  on  the  model  of  the  simple 
game.  In  this  form  the  decision  exercises  are  easy  to  play,  can  be  played  fairly  quickly,  are 
technologically  simple,  have  simple  rules,  are  flexible  and  adaptable,  and  are  very  transportable 
— they  should  be  capable  of  being  played  in  a  lunchroom  or  during  travel  layovers.  Each 
exercise  is  simple  because  it  focuses  on  only  one  or  a  few  particular  key  concepts,  principles,  or 
lessons  learned. 

If  a  decision  exercise  were  to  be  made  higher  fidelity,  it  could  be  thought  of  as  a  form  of 
simulation.  Simulations  at  some  level  of  fidelity  may  be  useful  in  utilities  training.  Computer- 
based  simulations  are  increasingly  being  used  to  teach  abstract  or  conceptual  information 
(Miller,  Eehman  &  Koedinger,  1999;  Resnick,  1994;  Schank  &  Earrel,  1988).  By  providing 
perceptual  grounding  of  abstract  concepts,  they  aid  comprehension,  and  the  interactivity  can 
support  a  deeper  understanding  of  the  material  (Goldstone  &  Sakamoto,  2003). 

Decision  Making  Exercises,  whether  the  brief  form  or  the  simulation,  have  some  general 
features: 

•  A  meaningful  title. 


168 


Report  on  FA8650-09-2-6033 


•  Background  information  describing  and  perhaps  explaining  the  current  situation  and  its 
origins, 

•  A  narrative  description  of  the  scenario  itself,  taking  the  form  of  a  compelling  story  that  builds 
to  a  climax — a  dilemma — putting  the  participants  on  the  hot  seat,  forcing  them  to  make  a 
decision  to  resolve  the  situation. 

•  Some  sort  of  visual  representation  that  accompanies  the  narrative  (diagram,  map,  etc.). 

•  If  meant  for  team  decision  making,  some  sort  of  role  assignments. 

Team  exercises  often  involve  an  energetic  exchange  of  opinions.  The  actual  decision  is  less 
important  that  the  thinking  that  goes  into  it.  The  decision  exercise  is  a  vehicle  for  triggering  the 
decision-making  process  and  then  allowing  individuals  to  reflect  on  it  or  discuss  it  with  others. 
Exercises  can  be  used  to  build  a  familiarity  and  mutual  understanding  within  a  team,  so  team 
members  know  more  about  how  others  are  likely  to  react  to  certain  types  of  situations. 

Exercises  come  from  personal  experiences,  which  are  described  as  narratives.  Exercises  are 
formed  around  specific  experiences  and  their  context.  Exercises  are  not  based  on  general  or 
typical  situations.  "We  need  context  in  order  to  exercise  our  intuition.”  (Klein,  2003,  p.  41)  Eor 
developing  instructional  cases,  domain  practitioners  are  generally  able  to  identify,  and  recall, 
previously  encountered  tough  cases.  In  addition,  managers  can  often  see  where  staff  members 
repeatedly  struggle  with  some  type  of  judgment  or  decision.  Both  the  practitioner  and  the 
manager  can  recount  personal  experiences.  Erom  these  one  can  create  a  timeline  for  the 
described  event,  and  for  each  key  decision  in  the  event,  a  list  of  the  information  requirements  of 
the  decision  maker.  Decision  Requirements  Tables  serve  the  dual  purpose  of  formulating  a 
description  of  the  type  of  dilemma  that  needs  to  be  practiced,  and  they  can  help  form  the  tough 
cases  to  be  use  in  decision  exercises. 

Another  benefit  of  construing  a  decision  exercise  as  a  game  is  that  one  can  hold  Tournaments.  In 
a  DMX  Tournament,  multiple  teams  (each  composed  of  a  few  to  a  handful  of  people,  all  having 
different  specializations)  each  craft  their  own  decision  exercise  and  then  all  the  teams  work  on 
the  exercises  of  the  others.  Then,  in  a  final  whole  group  discussion,  people  can  compare  notes  on 
what  was  frustrating  them,  and  what  made  exercises  challenging  for  others. 

Several  of  the  participants  planned  to  set  up  periodic  “field  days” 
during  long  lunch  breaks.  They  thought  that  routinely  having  their 
colleagues  make  up  and  play  decision  games  would  be  a  good 
chance  for  cross-functional  learning,  for  making  connections,  and 
for  building  intuitive  decision-making  skills.  Having  the  decision 
games  center  around  real  issues  meant  that  the  decision  games  were 
directly  relevant  to  their  work  and  provided  a  way  to  make  sense  of 
problems  that  were  troubling  them.  They  even  decided  they  could 
bring  the  games  to  upper  management  to  help  them  articulate  their 
frustrations  (Klein,  2003,  p.  42). 

Klein  (2004)  argues  that  Decision  Making  Exercises  can  help  individuals: 

•  Broaden  and  enrich  their  mental  models 

•  Establish  relationships  among  concepts  and  cases 

•  Appreciate  the  importance  of  critical  cues  and  patterns 

•  Eill  in  the  gaps  in  their  experience  base 

•  Eearn  ways  to  better  handle  uncertainty 
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•  Practice  at  resolving  conflicting  goals 

•  Learn  to  perceive  how  to  spot  leverage  points  -  the  starting  points  for  constructing  new 
option 

•  Learn  how  to  detect  and  anticipate  problems 

•  Understand  situations  from  alternative  perspectives 

•  Practice  at  allocating  limited  resources 

•  Acquire  technical  knowledge  more  quickly,  by  putting  it  in  a  practical  context 

•  Practice  at  giving  directions  or  presenting  clear  statements  of  assessments  or  intentions. 

The  Tournament  may  be  useful  for  individual  utilities  companies  but  may  also  be  run  at  the 
corporate  level,  if  one  were  to,  for  example,  hold  decision  exercises  for  Bulk  Power  Operations 
personnel  from  a  number  of  subsidiary  companies.  In  this  case  one  thinks  of  expertise 
acceleration  not  in  terms  of  being  faster  at  getting  an  individual  up  to  expert  levels  but  better  at 
spreading  the  corporate  expertise  around — getting  the  organization  as  a  whole  to  increase  the 
rate  at  which  it  shares  its  expert  knowledge  and  skill.  This  Tournament  notion  may  be  of  value  to 
utilities  that  seek  to  share,  as  well  as  accelerate,  their  expertise,  for  the  sake  of  promoting  the 
industry.  In  particular,  tournaments  may  be  of  value  in  generating  a  new  national-level  cohort 
prepared  to  launch  the  next  generation  of  nuclear  capabilities. 

5.5.8  Training  for  Teamwork 

The  study  of  team  cognition  and  coordination  is  relatively  new  (see  section  2.6,  above)  but  the 
research  has  made  clear  that  team  process  can  be  studied,  team  performance  can  be  measured, 
and  the  quality  of  teamwork  evaluated.  We  have  a  rich  corpus  of  concepts  with  which  to  make 
useful  distinctions:  task  work  vs.  team  tasks  vs.  teamwork;  team  vs.  individual  mental  models. 

Many  of  the  above  findings  and  ideas  about  acceleration  that  have  been  discussed  in  this  Report 
have  implications  for  training  for  teamwork  and  team  skills.  For  example,  much  of  the  training 
for  resilience  and  adaptivity  is  training  of  teams.  All  of  the  domains  that  have  been  discussed  in 
this  Report  involve  teaming  and  therefore  a  need  for  team  training  (see  Table  4.1,  above).  All  of 
the  larger-scale  exercises  we  have  discussed  (the  NTC,  Operational  Simulation,  etc.)  are  methods 
that  train  teams  at  the  same  time  they  are  training  individuals  Time  compression  can  apply  to 
simulated  cases  involving  problem  solving  by  teams,  and  not  just  individuals.  Decision  Making 
Exercises  and  Anticipatory  Training  exercises  involve  problem  solving  and  discussion  by  teams, 
and  not  just  individuals. 

In  many  military  jobs  and  missions,  military  people  do  not  stay  together  in  teams.  It  is  hard  to 
keep  people  in  the  military  together  as  people  get  transferred.  War  has  its  own  effects  that  work 
against  the  notion  of  building  expert  teams.  Training  for  high  levels  of  team  proficiency  must 
involve  training  in  team  tasks  in  addition  to  training  in  the  teamwork. 

•  An  outstanding  challenge  is  “how  to  turn  a  team  of  experts  into  an  expert  team,”  how  to 
turn  a  new  team  into  a  high-functioning  team. 

•  An  outstanding  challenge  is  how  to  train  for  team  skills  such  that  a  set  of  proficient 
individuals  can  rapidly  compose  itself  ad  hoc  to  form  a  high  proficiency  team. 

•  An  outstanding  challenge  is  to  ask  “the  hiatus”  question  concerning  teams.  In  what  ways 
might  team  skills  decay  due  to  hiatus  experiences. 
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•  An  outstanding  challenge  is  how  to  conduct  knowledge  management  for  team  skills,  that 
is,  the  capture,  preservation  and  reuse  of  knowledge  and  skills  specifically  with  regard  to 
team  tasks  and  teamwork. 

Attempts  at  acceleration — in  any  of  its  four  senses — could  contribute  to  our  ability  to  promote 
teamwork  and  team  training  by  advancing  theory  and  method  in  the  study  of  team  cognition  and 
collaboration. 


5.6  Facilitated  Retention  of  Skill 


5.6. 1  Opportunity:  Reserves  Training 

It  might  be  possible  to  establish  a  research  project  in  collaboration  with  Reserve  Units, 
representing  an  opportunity  to  study  retention  and  retraining. 

5.6.2  Targeted  Research  Need:  Measuring  Retention/Decay  and 

Hiatus  Effects 

There  is  a  consensus  on  two  outstanding  needs.  One  need  is  for  better  measures  of  over-learning. 
Proficiency  can  and  does  continue  to  increase  even  after  performance  achieves  some  metric  for 
competence  (minimal  or  otherwise)  on  any  given  measure  of  performance.  For  example,  air 
traffic  controllers  continue  improving  their  skills  long  after  they  achieve  a  criterion  of  “no 
errors”  in  the  simulation-based  training.  We  know  that  practice  trials  or  time  in  practice  only 
partially  correlates  to  increasing  proficiency.  This  is  as  likely  as  true  for  performance  after 
achieving  some  criterion  as  it  is  for  performance  before  achieving  some  criterion.  On  the  other 
hand,  with  more  time  in  performance  there  is  an  increasing  chance  of  encountering  tough  cases, 
which  also  contribute  to  increases  in  proficiency. 

In  some  domains,  experts  make  very  few,  if  any  errors.  Examples 
are  air  traffic  controllers,  short-term  weather  forecasters,  insurance 
actuaries,  photo  interpreters  Experts  keep  improving  and  thus 
“percent  correct”  is  insensitive.  How  can  we  assess  performance 
that  keeps  improving?  We  need  to  compare  various  measures  for 
evaluating  behavior  that  is  “better  than  perfect”  (Shanteau,  et  ah, 

2002). 

One  method  that  is  suggestive  of  new  possibilities  is  the  Cochran-Weiss-Shanteau  Index  (CWS). 
This  index  is  reminiscent  of  the  notions  of  signal  detection  theory  in  that  it  is  formed  as  a  ratio  of 
discrimination  to  inconsistency.  Einer  discrimination  and  less  inconsistency  entail  better 
performance.  It  is  a  relative  index  (therefore  not  bounded  as  are  singular  measures  of  positive 
performance  (percent  correct)  or  errors.  It  is  nevertheless  sensitive  to  errors,  more  so  than  peer 
judgments  of  performance  (Thomas,  et  ah,  2001).  Also,  values  can  only  be  compared  within 
domains.  This  and  other  measures  might  be  brought  to  bear  in  studies  of  high  proficiency  in 
military  domains,  as  Eriel,  et  al.  (2002)  and  Thomas,  et  al.  (2001)  have  shown  for  the  case  of  air 
traffic  control. 
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5.6.3  Targeted  Research  Need:  Hiatus  Training  to  Prevent  Decay 

With  regard  to  issues  of  decay/retention,  especially  following  hiatus,  loss  of  skill  on  the  part  of 
pilots  has  been  the  prototype  for  discussion.  Many  considerations  enter  in.  Loss  of  skill  due  to 
hiatus  will  likely  continue  to  be  a  problem  as  the  Air  Force  makes  trade-offs  between  capabilities 
of  warfighting  and  capabilities  of  administration.  Some  performance  data  show  only  modest  loss 
of  skill  and  reasonably  rapid  reacquisition  of  skill  following  hiatus.  Piloting  and  training  have 
been  studied  extensively,  but  for  acceleration  research  there  would  be  a  need  for  new  measures 
of  performance,  over-learning,  and  proficiency  level.  On  the  other  hand,  semi- standardized 
programs  for  refresher  training  represent  tradition,  are  entrenched  in  both  the  Air  Force  and  the 
Navy,  and  are  ipso  facto  adequate.  Another  paradox  at  play  is  the  fact  that  duty  assignments 
during  hiatus  generally  leave  little  time  for  practice  or  refresher  training. 

Social  Networking  might  play  a  role,  permitting  pilots  to  keep  current  on  checklists,  procedures, 
and  simulations  of  various  kinds  and  contacts  with  active  flyers  (e.g.,  through  blogs)  for  keeping 
up  with  tactics  and  equipment  advances. 

Psychological  research  has  shown  that  “mental  practice,”  which  is  practice  within  the  medium  of 
thought,  imagery  and  the  kinesthetic  sense,  can  lead  to  performance  improvement.  If  viable, 
mental  practice  might  play  a  role  in  hiatus  training  for  skill  retention.  Since  in  the  hiatus  situation 
it  might  be  inconvenient  or  impossible  to  practice  a  skill,  especially  if  the  skill  requires  special 
equipment  not  readily  available.  The  question  arises  as  to  whether  mental  practice  can  provide  a 
reasonable  substitute  for  actual  practice  in  those  circumstances.  In  one  study  using  a  data  entry 
task,  participants  practiced  typing  four-digit  numbers  by  actually  typing  or  by  just  imagining  the 
typing  movements.  Mental  practice  participants  showed  improvements  in  typing  skill  equal  to 
that  for  the  typists.  When  practice  involved  a  key  configuration  different  from  that  used  at 
testing,  participants  who  used  physical  practice,  but  not  those  who  used  mental  practice,  suffered 
from  interference  (Healy,  2007). 

5.6.4  Targeted  Research  Needs:  Rapidized  Updating 

“Rapidized  Updating”  would  leverage  an  institutionalized  knowledge  management  process  to 
rapidly  transition  workplace  changes  into  a  mockup  (of  suitable  fidelity)  and  thereby  enable 
pilots  to  practice  at  what  has  changed  in  their  work  context.  This  new  concept  might  be 
investigated  in  collaboration  with  DoD  Programs  that  are  currently  focusing  on  the  notion  of 
accelerating  the  transition  of  lessons  learned  from  the  battlespace  into  the  training  space. 


5.7  Accelerated  Achievement  of  Proficiency 

We  begin  this  final  section  of  this  Report  by  presenting  vignettes  based  on  the  responses  of 
Working  Meeting  participants  to  the  core  questions  of  the  meeting. 


5. 7. 1  Four  Vignettes 

Vignette  #1:  Task  Analytic/Factorial  Experiment  Approach 

I  would  take  an  analytical  experimental  approach  to  the  issue  of 
accelerating  the  achievement  of  proficiency  in  some  selected 
domain.  In  particular,  I  would  start  by  breaking  down  the  required 
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tasks  in  the  selected  domain  into  component  subtasks.  I  would  then 
conduct  experiments  that  isolate  the  various  subtasks  to  test  specific 
hypotheses  about  how  to  optimize  not  just  the  efficiency  of  learning 
(speed)  but  also  its  durability  (retention  across  delays)  and 
generalizability  (transfer  from  one  set  of  circumstances  to  another). 

In  particular,  I  would  conduct  experiments  to  test  whether  training 
principles  developed  in  my  previous  research  apply  to  these 
component  subtasks.  For  example,  I  would  test  whether  the 
procedural  reinstatement  principle  applies  to  the  subtasks  in  this 
domain.  By  this  principle,  for  procedural  information  (skill 
learning),  retention  is  strong  but  transfer  is  limited,  whereas  for 
declarative  information  (fact  learning),  retention  is  poor  but  transfer 
is  robust.  For  another  example,  we  have  been  able  to  improve  fact 
learning  to  a  huge  extent  by  using  the  strategic-use-of-knowledge 
principle,  by  which  learning  and  memory  are  facilitated  whenever 
pre-existing  knowledge  can  be  employed  as  a  mediator  in  the 
process  of  acquisition.  I  would  test  whether  that  principle  might 
also  be  used  to  accelerate  the  learning  of  subtasks  in  the  selected 
domain.  I  would  further  hope  to  conduct  experiments  aimed  to 
develop  new  training  principles  for  accelerating  the  learning  of 
subtasks  in  the  selected  domain  as  well  as  for  enhancing  the 
retention  and  transfer  of  the  learned  subtasks. 

Minority  Report:  While  task  analysis  of  a  number  of  forms  will  certainly  be  a  component  of 
research  on  acceleration,  an  assumption  of  this  vignette  is  that  the  analysis  will  be  of  fixed  tasks. 
A  second  assumption  is  that  the  demonstration  of  acceleration  is  identical  to  controlled 
laboratory  experimentation  on  familiar  effects  and  principles.  While  both  assumptions  define  a 
valid  research  space,  they  are  bounded  when  seen  in  the  context  of  complex  cognitive  work 
systems,  which  have  changing  and  emergent  goals  typifying  the  evolving  work  of  the  military. 

Vignette  #2:  Train  Maintenance  Tasks  Using  Augmented  Reality 

If  I  had  an  opportunity  to  accelerate  the  achievement  of  proficiency 
in  a  selected  domain.  I’d  first  select  a  domain  that  would  be  relevant 
to  military  requirements  and  that  would  afford  a  cost-effective 
opportunity  for  experimental  work.  For  these  reasons.  I’d  probably 
pick  maintenance  as  the  domain  since  it  is  common  across  many 
fields  and  civilian/military  environments,  it’s  subject  to 
technological  advancement  and  knowledge/skill  loss,  something 
that  could  be  studied  in  an  academic  lab,  is  suitable  for  performance 
measurement  with  little  additional  instrumentation,  and  also 
suitable  to  experimental  manipulation  (e.g.,  number  of  programmed 
faults  or  their  probability).  The  potential  subject  populations  of 
novices  and  experts  could  be  large  if  say  auto  maintenance  was 
chosen  as  the  operational  context,  another  advantage.  I’d  then 
choose  independent  variables  that  might  yield  the  most  bang  for  the 
buck,  and  consider  not  only  variables  that  might  promote  learning 
but  also  ones  that  seem  to  be  holding  it  back,  such  as  sleep  loss  and 
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interrupted  learning  or  practice  (due  to  multitasking  within  the 
military).  As  a  favorite,  I’d  consider  how  new  technology,  such  as 
augmented  reality,  could  be  used  in  both  a  schoolhouse  and  a 
deployed  setting  (i.e.,  as  part  of  an  embedded  training  system  in 
operational  equipment).  I’d  choose  this  technology  for  study 
because  many  nations  are  moving  to  embedded  training,  but  little  is 
known  about  its  efficacy.  Augmented  (mixed,  fused,  VR  etc.)  are 
potential  enablers  of  embedded  training.  It  makes  sense  to  me  to 
investigate  potential  technologies  or  interventions  that  could  apply 
in  an  operational  military  setting  where  the  benefits  could  be  large 
(e.g.,  mechanics  having  to  learn  quickly  how  to  diagnose  and  repair 
a  new  piece  of  kit  introduced  directly  to  the  field).  I’d  want  to 
compare  learning  rate,  transfer  effectiveness,  and  retention 
(different  intervals)  associated  with  this  approach  to  conventional 
training  methods. 

Minority  Report:  The  selection  of  a  domain  having  military  and  civilian  instantiations  is  a  key 
aspect  of  this  vignette.  If  a  domain  such  as  this  were  chosen,  a  concerted  effort  would  be 
required  to  avoid  any  focus  on  fixed  tasks.  The  main  idea  for  acceleration  is  virtual  reality  and 
embedded  or  on-the-job  training.  A  current  analog  would  be  the  EPRI  use  of  VR  to  train  for 
maintenance  in  a  nuclear  power  plant's  "hot  room." 

Vignette  #3:  How  Trainers  Do  It  Now 

I’ll  argue  that  in  our  projects  we  already  accelerate  the  achievement 
of  proficiency,  in  our  simulation  training  systems.  Which  is  not  to 
say  we  cannot  improve  the  process.  The  process  is  not  probably 
conceptually  different  from  many  others.  It  requires:  subject-matter 
experts;  critical  tasks;  critical  skills  (note,  we  focus  on  skills,  not 
knowledge)  that  underlie  critical  tasks;  performance  measures  for 
critical  skills;  and  specific  actions  (sequenced,  when  necessary)  that 
define  proficient  performance.  We  model  the  student’s  current  skills 
proficiency  and  to  devise  the  most-appropriate  next  lesson  in  crawl- 
walk-run  fashion  to  address  something  like  the  student’s  zone  of 
proximal  development.  So  what  is  it  that  is  in  any  way  special?  I’ll 
put  forth  six  possibilities,  with  one-sentence  descriptions  doing  no 
justice  to  the  lengths  of  papers  to  which  each  refers. 

1 .  Scenario-based  training.  The  student  acquires  and  practices  skills 
to  the  extent  reasonable  (not  “the  extent  possible,"  since  it  depends 
on  cost-effectiveness)  in  a  realistic  setting. 

2.  Scenario-based  assessment.  The  student  demonstrates  skills  in  a 
situation,  the  student  does  not  tell  you  what  s/he  would  do  or 
believes  is  appropriate  in  a  situation. 

3.  Parameterized  dynamic  scenarios.  To  the  extent  reasonable,  build 
the  scenario  for  the  next  lesson  based  on  all  that  is  known  about  the 
student,  the  student’s  current  skills  deficiencies,  and  the  student’s 
recent  learning  history. 
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4.  Imperfect  conceptual  models.  Try  to  entice  failure,  then  provide 
the  structure  (e.g.,  explanations  of  actions  in  context)  to  allow  the 
student  to  engage  in  reflection  so  as  to  confront  biased  or  erroneous 
thinking. 

5.  Embodied  tutoring.  Present  a  face  to  a  tutor,  potentially  an  avatar 
with  the  knowledge  to  vary  characteristics  such  as  personality  and 
current  emotion,  level  of  ‘intrusiveness’,  and  tutoring  style. 

6.  Augmented  training.  Monitor  as  much  as  is  reasonable  of  the 
student,  to  include  overt  actions,  her/his  expressions  and  gestures, 
eye  movement,  affect,  physiologic  indicators,  neurologic  function, 
and  others. 

Minority  Report:  This  proposal  captures  many  of  the  ideas  discussed  at  the  Working  Meeeting: 
reliance  on  experts  as  the  "gold  standard,"  scenario-based  training  and  augmented  (intelligent) 
tutoring. 

Vignette  #4:  Leveraging  Known  Generalizations 

The  first  principle  is  that  if  you  want  rapid  and  accurate 
performance,  the  trainee  needs  to  have  relevant  patterns/cases 
highly  accessible.  Development  of  a  case  corpus  for  familiar  cases 
can  be  accomplished  since  those  cases  are  encountered  frequently 
or  recently.  Many  domains  have  (perceptual)  features  that  vary  in 
occurrence  from  frequent  to  rare  but  are  critical  to  carrying  out 
sound  actions  in  the  domain.  It  is  relatively  easy  to  pick  up 
frequently  occurring  features/cues  with  most  training  techniques. 

Rare  cases,  by  definition,  occur  infrequently,  so  one  way  to 
accelerate  development  of  skill  is  to  flatten  the  curve  by  presenting 
such  cases  as  part  of  training.  However,  the  "shape"  of  the  task 
environment  becomes  important  here.  If  rare  cases  don't  have 
important  consequences,  but  frequent  cases  do,  it  is  not  worth 
investing  a  lot  of  effort  in  training  rare  cases  compared  to  more 
frequent  (important)  ones.  Power  law  learning  means  that  each 
increment  in  skill  demands  a  log  unit  of  investment/practice.  Again, 
the  shape  of  the  task  environment  becomes  critical  given  the  law  of 
diminishing  returns  for  training  (another  way  to  phrase  power  law 
learning).  I'm  assuming  that  training  cost  is  constant  per  unit  (hr.)  of 
training.  So,  training  efforts  need  to  focus  on  getting  someone  up  to 
speed  and  then  after  they  reach,  say,  journeyman  skill  levels,  to 
maintaining  performance  via  refresher  training  to  ensure  that 
recency  can  work  its  magic  for  rapid  access. 

Minority  Report:  This  vignette  captures  additional  ideas  discussed  at  the  Working  Meeting, 
including  the  need  for  a  case  corpus.  This  vignette  also  points  to  domains  where  perceptual 
learning  is  critical.  Especially  salient  is  the  idea  of  accelerating  learning  by  accelerating  the 
specific  part  of  the  proficiency  continuum  leading  up  to  the  journeyman  level. 

5. 7.2  Research  Opportunity:  Rapidized  Training 
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It  might  be  possible  to  establish  a  research  project  in  collaboration  with  one  or  more  of  the 
current  DoD  efforts  at  rapidized  training,  such  as  the  Air  Force  Weather  Agency's  "Forecast 
Ready"  program  (McNulty,  2005)  and  "Cadre  Focus"  program  (Smith,  2006),  and  the  "Think 
Like  a  Commander"  program.  The  7th  Weather  Squadron's  "Cadre  Focus"  initiative  can  be  seen 
as  following  the  model  of  the  Army's  National  Training  Center.  Its  focus  is  on  making  sure  that 
forecasters  are  tactically  and  technically  prepared  prior  to  actual  deployment.  The  method  is  to 
site  students  at  a  large  training  area  for  a  week  of  instructions  (vehicle  maintenance,  generator 
maintenance,  simulated  convoy  operations,  etc.)  and  experience  with  command  and  control 
procedures,  in  addition  to  practice  at  tactical  meteorological  forecasting. 

It  might  be  possible  to  establish  a  collaboration  with  the  U.  S.  Army  Corps  of  Engineers.  The 
249th  Engineer  Battalion,  based  at  Et.  Belvoir  VA,  provides  power  generation  for  all  of  the  U.S. 
Army.  Although  the  U.S.  Air  Eorce  has  its  own  construction-power  generation  specialist  group, 
the  Army  Battalion  also  works  with  them.  It  has  about  200  specialists  in  a  number  of  areas  (e.g., 
generation,  instrumentation,  electronics).  The  Battalion  supports  a  Civil  Engineer  Maintenance, 
Inspection  and  Repair  Team  that  is  capable  of  installing  and  operating  commercial-grade  power 
generation  facilities  (e.g.,  in  support  of  airfields  and  base  camps).  (They  also  oversee  the  work  of 
commercial  contractors  who  install  and  operate  larger-scale  power  systems.) 

Most  of  the  specialists  are  regarded  as  experts,  though  naturally  the  Battalion  has  trainees  at 
various  levels  of  skill  and  knowledge.  The  schoolhouse  training  is  1.4  years,  with  high  math  skill 
as  a  selection  criterion.  Once  at  the  Battalion,  there  is  significant  mentoring  on  various  power 
generation  systems,  increasing  the  complexity  using  a  "crawl-walk-run"  model.  Eore  is  that  it 
takes  three  to  five  years  for  a  person  to  become  a  journeyman  and  ten  or  more  years  to  become 
regarded  as  a  master  electrician.  The  Battalion  Command  desires  to  have  half  of  its  personnel 
complement  at  the  expert  level  of  proficiency. 

Although  specialists  generally  work  in  their  specialty  area  throughout  their  career,  there  are 
hiatus  assignments  (recruiting,  doctrine,  etc.).  On  occasion  up  to  one-fourth  of  the  complement 
of  specialists  is  on  assignment,  but  the  assignments  involve  power  generation  and  distribution 
issues  (e.g.,  providing  back-up  emergency  power).  Eore  is  that  it  takes  up  to  three  months  to  "get 
back  up  to  speed"  following  a  shorter  hiatus  of  a  year  or  so.  Hiatus  assignments  of  longer  periods 
(three  years)  are  not  unheard  of,  but  are  infrequent.  There  has  been  no  formal  study  of  hiatus 
effects.  Also  hearkening  to  the  Air  Eorce  experience,  workload  during  hiatus  assignments  is  such 
that  it  would  be  difficult  for  the  specialist  to  maintain  currency  (i.e.,  aware  of  the  latest  changes 
in  power  technology  and  Army  power  systems  and  procedures).  The  Battalion  does  host  a  web 
site,  which  could  assist  in  knowledge  sharing  of  lessons  learned.  The  Command  is  open  to 
collaboration  to  research  new  and  non-traditional  methods.  Chief  of  Engineers  General  Robert  E. 
Van  Antwerp  has  discussed  the  Corps's  level  of  risk  at  loss  of  expertise,  mirroring  the  knowledge 
management  issue  faced  by  many  other  domains  and  organizations. 

5.7.3  Research  Challenge:  Methodology  for  Identifying  Expert 
Mentors 

We  have  a  good  idea  of  the  characteristics  of  good  mentors  (Hoffman,  1998):  Klein,  1998;  Eiore 
&  Salas,  2008;  Mistrell,  1999;  Proctor  &  Vu,  2006  (see  also  Section  2.3.5  above).  There  is  often 
motivation  of  experienced  workers  to  pass  along  their  "tough  case"  knowledge,  but  they  are 
typically  left  not  knowing  how  to  proceed,  or  not  even  knowing  that  their  "tough  case" 
knowledge  is  critical  and  highly  constructive  to  apprentices. 
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The  expert  mentor  can  play  an  active  role  not  only  in  on  the  job  learning,  but  in  guided  learning 
as  would  take  place  in  simulation-based  training  episodes.  This  could  be  implemented  with 
simulation-based  environments  of  the  sort  that  have  been  discussed  in  this  Report  (i.e.,  Collins, 
Brown,  &  Newman,  1989;  Schaper  &  Sonnentag,  1998). 

Given  the  critical  role  of  the  mentor,  and  the  potential  for  mentoring  to  promote  advancement  in 
proficiency,  efforts  at  acceleration  would  be  supported  in  a  significant  way  by  the  development 
of  a  method  for  rapidly  identifying  individuals  within  a  domain  who  are  great  mentors.  We  know 
how  to  go  about  and  cognitive  task  analysis  to  reveal  their  mentoring  reasoning  strategies, 
including  strategies  for  dealing  with  challenging  mentoring  situations.  But  there  is  currently  no 
generally  applicable  and  robust  method  for  entering  into  a  domain  and  rapidly  identifying  who 
is,  and  who  might  become,  a  good  mentor. 

5.7.4  Research  Concept:  Compression  of  Experience 

It  is  well  known  that  extensive  experience  with  challenging  cases  and  deliberate  practice  (and 
guidance)  are  hallmarks  of  the  development  of  expertise.  One  likely  reason  for  the  "10,000 
hours"  guideline  for  the  development  of  expertise  is  that  ordinary  work  flow  does  not  provide 
enough  exposure  to  "tough  cases,"  those  that  stretch  skill,  and  does  not  present  them  within  any 
kind  of  logical  structure  for  learning.  Finding  ways  to  compact  experiences  will  be  vital  to  any 
program  aiming  to  accelerate  expertise.  Methods  might  include  games,  simulations,  virtual 
worlds,  decision-making  exercises,  and  other  kinds  of  case-oriented  delivery  systems  (e.g., 
Spiro,  Thota,  &  Feltovich,  2003). 

Compression  would  be  achieved  because  the  naturally  occurring  time  lags  and  gaps  between 
cases  that  afford  learning  opportunities  could  be  “designed  out.”  The  caution  is  that  experience 
of  multiple  difficult  cases  in  an  artificially  compressed  time  period  might  contribute  to  fatigue 
and  other  negative  affects.  Oh  the  other  hand,  some  jobs  entail  stress  and  fatigue  so  this  might  be 
less  of  a  factor. 

Tough  and  rare  cases  can  also  be  used  in  unlearning,  to  reveal  knowledge  shields  and  help 
trainees  overcome  "dimensions  of  difficulty"  within  cases  (Feltovich  et  ah,  1997).  In  particular, 
the  learner  must  be  constantly  stretching  his  or  her  skill,  going  beyond  the  current  level  of 
comfortable  competence.  Rare  cases  would  be  juxtaposed  with  some  routine  or  typical  cases  in 
such  a  way  that  the  learner  can  experience  the  differences  that  produce  difficulties.  Another 
application  involves  the  fact  that  the  sociotechnical  workplace  is  constantly  changing  (Hoffman 
&  Elm,  2006).  New  technologies,  and  the  work  methods  that  they  shape,  typically  have  to  be 
integrated  with  legacy  work  methods  and  technologies.  Conceptually,  this  can  be  viewed  as  a  set 
of  scenarios  for  practice  at  adaptation,  not  merely  to  handle  familiar  and  routine  cases,  but  to 
"stretch"  the  new  technologies  and  work  methods  through  application  in  tough  cases. 

If  one  accepts  the  finding  from  studies  of  experts  that  ten  years  is  a  lower  limit,  and  that  a 
lengthy  period  of  practice  with  zeal  is  necessary,  then  the  best  that  organizations  could  hope  for 
is  to  “pre-burn”  some  few  of  the  years  during  the  novice-to-apprentice  and  apprentice-to- 
journeyman  spans  of  training.  As  is  shown  in  Figure  5.1,  this  would  accelerate  the  “front  end”  of 
the  learning  curve  that  occurs  prior  to  the  long  ramp  in  the  high  proficiency  range.  We  need  not 
be  trapped  by  the  “10,000  hour  clock”  if  the  focus  were  on  accelerating  proficiency  between  the 
apprentice  and  journeyman  levels. 
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Figure  5.1.  Possible  proficiency  achievement  and  acceleration  curves. 


Let’s  say  the  red  curve  represents  the  learning  trajectory  for  some  domain.  In  the  red  curve  it 
takes  nearly  50%  of  the  overall  learning  time  to  reach  journeyman  status  in  that  domain.  A 
notional  large  leap  occurs  during  the  apprentice-to-senior  journeyman  range  stage.  And  there  is  a 
long  ramp,  as  many  researchers  in  the  field  of  Expertise  Studies  have  surmised.  The  green  curve 
is  one  where  some  intervention  is  introduced  (e.g.,  time  compressed,  simulation-based  training). 
Here  it  takes  approximately  a  third  of  the  overall  learning  time  to  reach  high  proficiency  status. 
But  the  long  ramp  is  still  present  and  what’s  happened  is  that  the  leap  in  proficiency  has  occurred 
earlier. 

Given  the  value  of  high  proficiency,  and  the  increasing  value  of  high  proficiency  in  emerging, 
dynamic  and  complex  domains,  if  any  time  could  be  shaved  off  the  “10,000  clock,”  there  would 
be  a  potentially  significant  savings  of  time,  an  increase  in  overall  organizational  capability,  and 
perhaps  also  an  increase  in  organizational  adaptability. 
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An  initial  project  would  have  to  gauge  the  time  compression  effects  themselves.  Since  it  is  well 
established  that  development  of  skill  in  any  complex  domain  takes  a  very  long  time,  we  are 
particularly  interested  in  seeing  if  practice  at  tough  cases  can  help  accelerate  the  development  of 
high  proficiency,  spanning  the  senior  apprentice  to  senior  journeyman  levels.  Thus,  performance 
at  the  tough  case  tasks  will  have  to  be  studied  in  a  short-term  longitudinal  design.  We  will 
compare  performance  at  tough  task  cases  for  individuals  who  receive  multiple  practice  sessions 
over  time,  to  individuals  who  are  sampled  across  time/proficiency  level,  in  an  effort  to  determine 
if  practice  has  a  “time  compression”  effect.  This  will  be  gauged  in  light  of  performance  at  cases 
that  are  routine. 


5.7.5  Study  Design 

We  conclude  by  considering  the  design  of  research  projects  that  would  contribute  to  attempts  to 
accelerate  accelerated  learning,  likely  focusing  on  the  progression  for  journeyman  to  senior 
journeyman,  or  from  senior  journeyman  to  junior  expert.  In  their  book  Surpassing  Ourselves, 
Carl  Bereiter  and  Marlene  Scardamalia  (1993)  note  the  necessity  for  studying  the  acquisition  of 
proficiency  longitudinally,  and  also  note  the  difficulties  of  doing  that: 

The  ten  years  or  so  of  training  and  experience  that  typically  go  into 
the  making  of  an  expert  pose  a  serious  obstacle  to  research  on  the 
acquisition  of  expertise.  The  obvious  kind  of  research  to  do  is 
longitudinal  and  prospective:  Follow  a  large  group...  through  their 
early  years  of  practice,  collecting  data  that  could  later  be  analyzed 
for  clues  as  to  why  some  achieve  high  levels  of  expertise  while 
others  become  experienced  non-experts.  But  no  one  funds  research 
of  that  duration  anymore.  And  it  would  have  to  be  intensive 
research,  tracking  people  in  the  course  of  their  work.  Merely 
giving  batteries  of  tests  and  correlating  scores  with  later 
performance  might  yield  findings  useful  for  screening  candidates, 
but  it  could  not  be  expected  to  add  much  to  our  understanding  or  to 
leads  for  improving  the  education  and  nurturance  of  experts  (p. 

155). 

Cohort-select  designs  will  study  multiple  groups,  each  of  which  has  members 
proficiency  level:  junior  apprentice,  apprentice,  senior  apprentice,  junior 
journeyman,  senior  journeyman,  and  junior  expert.  (As  explained  previously,  the  focus  would 
not  be  on  expertise  per  se,  but  participation  by  experts  would  be  critical  to  the  creation  of  both 
proficiency  scales  and  training  materials.)  Each  of  the  groups  is  tracked  across  time.  Proficiency 
scaling  would  identify  individuals  who  learn  fast  and  slow.  Retrospective  analyses  would  seek  to 
find  out  why.  One  could  identify  the  top  performers  at  each  proficiency  level  (test  scores,  records 
of  past  performance  during  training)  and  work  backwards  to  find  the  success  factors.  One  could 
study  acquisition  retrospectively  and  study  degradation  and  retraining  prospectively  and  thereby 
“cover”  more  of  the  proficiency  span  than  the  five  to  eight  year  duration  of  the  research  project 
itself. 

Suggestions  for  acceleration  methods,  as  discussed  previously  in  this  Report,  include  leveraging 
the  important  features  of  deliberate  practice  and  mentoring,  the  technologies  of  virtual  reality 


that  fall  at  a 
journeyman. 
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training,  multi-media  for  training  at  complex  tasks,  and  leveraging  computer  technology  to 
develop  libraries  of  representative  and  tough  tasks  and  knowledge  bases  that  capture  expert 
knowledge  and  skill. 

Proficiency  scaling  conducted  as  a  part  of  a  longitudinal  study  might  also  contribute  to  selection. 
Bereiter  and  Marlene  Scardamalia  (1993)  noted  that  the  practice  habits  and  learning  strategies 
that  culminate  in  the  achievement  of  high  levels  of  proficiency  are  not  just  characteristics  of 
experts,  but  are  also  the  characteristics  of  trainees  who  will  eventually  become  experts. 

Lacking  long-term  research  that  follows  the  same  people  from 
novicehood  to  mature  status,  we  are  left  having  to  look  for 
plausible  resemblances  or  parallels  between  what  may  be  observed 
among  students  and  what  is  known  about  experts.  It  is  therefore  a 
matter  of  considerable  note  that  students  can  be  found  who  already 
resemble  experts  in  important  respects.  Case  studies  provide  no 
basis  for  estimating  how  numerous  such  students  are,  but  the  fact 
that  they  exist  at  all  should  give  us  something  to  ponder  (pp.  154- 
155). 

The  government  sponsor  of  this  work  could  ground  the  research  program  in  a  challenge  problem 
(assuming  the  potential  diversity  of  stakeholders  permits  this).  That  challenge  problem  should 
specify  a  domain  or  criteria  for  selection  of  a  domain,  a  statement  of  the  desired  operational 
objectives,  and  related  measures  of  performance  for  assessing  research  (though  not  necessarily 
performance  thresholds).  Research  efforts  should  focus  on  approaches  that  enable  DoD  to 
specify  a  balance  between  accelerating  acquisition  and  facilitating  retention.  Research  should  be 
presented  at  semi-annual  sponsor-hosted  workshops.  These  workshops  should  evolve  in  topic 
from  description  of  research  methods,  to  reports  of  results,  and  finally  to  presentation  of 
technology  or  materials  that  enable  DoD  or  its  performers  to  implement  the  research  or  training 
methods  in  new  domains. 

The  sections  5.1  through  5.7  of  this  Report  presented  recommendations  concerning  the  study  of: 
(1)  Measures  and  levels  of  proficiency,  (2)  Selection  and  scaffolding  for  expertise  development. 
(3)  Mentoring  and  apprenticeship,  (4)  Practice  and  experience  compression,  (5)  Learning  for 
flexibility,  adaptability,  and  resilience,  (6)  Procedures  to  improve  retention,  and  (7)  All  of  these 
factors  in  the  context  of  teams  and  groups.  Based  on  these  recommendations,  it  is  possible  to 
formulate  a  roadmap  for  investigations  in  accelerated  proficiency  and  facilitated  retention.  Such 
a  roadmap  would  refer  to  all  of  the  meanings  of  accelerated  learning: 

1)  How  to  quicken  the  training  process  while  maintaining  its  effectiveness  (Rapidized 
Training), 

2)  How  to  rapidize  the  transposition  of  lessons  learned  from  the  battlespace  into  the  training 
context  {Rapidized  Knowledge  Sharing), 

3)  How  to  train  and  train  quickly  to  higher  levels  of  proficiency  {Accelerated  Proficiency), 
and 

4)  How  to  insure  that  training  has  a  stable  and  lasting  effect  {Facilitated  Retention). 
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A  notional  roadmap  is  presented  in  Figure  5.2.  This  timeline  lays  out,  over  months/years,  a 
sequenced  set  of  research  activities  for  demonstrating  accelerated  learning  and  facilitating  long¬ 
term  retention.  Proposals  to  conduct  studies  would  start  from  this  high-level  roadmap  and  forge  a 
detailed  program  description,,  taking  into  account  the  chosen  domain(s),  opportunities,  practical 
constraints  and  considerations.  Planning  details  would  morph  as  the  project  is  constrained. 

Figure  5.2  A  notional  roadmap  for  studies  on  accelerated  proficiency  and  facilitated  retention 
(page  following). 
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Year  1 

Year  2 

Year  3 

Year  4 

Year  5 

Year  6 

Year  7 

6  12 

18  24 

30 

36 

42 

48 

54 

60 

66 

72 

78 

84 

Methodological  Foundation  and  Requirements 

Select  domains  and  establish  links  with  domain  organizations  and  cohorts  of 
practitioners 

Integrate  with  the  training  community  in  the  selected  domain 

Career  interviews,  performance  measures,  and  sociometry  for  proficiency  scaling 

Documentation  analysis  (i.e.,  SOP  documents,  task  analyses,  syllabi)  to  generate 
normative  models  of  reasoning  and  models  of  knowledge 

Cognitive  Task  Analysis  to  generate  models  of  reasoning  and  models  of  knowledge 

Cognitive  Task  Analysis  to  generate  case  corpus 

Leverage  Cognitive  Task  Analysis  to  generate  measures  and  metrics  for  performance 

Identify  and  study  journeyman/expert  mentors 

Retrospective  Study 

Identify  cohort  of  experts  and  retired  experts 

Career  analyses  to  track  performance  and  learning  gains 

Table  continues 
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Year  1 

Year  2 

Year  3 

Year  4 

Year  5 

Year  6 

Year  7 

6 

12 

18 

24 

30 

36 

42 

48 

54 

60 

66 

72 

78 

84 

Accelerated  Proficiency 

Refine  the  "cohort-select"  experimental  design 

Refine  the  plan  for  the  training  and  testing  protocols 

Planning  to  integrate  the  research  activities  with  the  current  training  in  the  domain 

Design  of  training  methods/exercises  (e.g.,  Time  Compression,  OPSIM,  DMX,  etc.) 
(see  Section  5) 

Identify  control  and  experimental  cohorts 

Baseline  performance 

Conduct  prospective  study 

Tests  for  robustness,  resilience,  adaptivity 

Evaluate  and  integrate  results 

Optional  tracking  to  verify  proficiency  achievement  level 


Table  continues 
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Year  1 

Year  2 

Year  3 

Year  4 

Year  5 

Year  6 

Year  7 

6 

12 

18 

24 

30 

36 

42 

48 

54 

60 

66 

72 

78 

84 

Facilitated  Retention 

Develop  measures  for  decay  and  retention  effects 

Design  of  maintenance/refresher  training  methods 

Design  modules/exercises 

Refine  the  experimental  design  (control  and  hiatus  groups) 

Identify  cohorts 

Baseline  performance 

Train  to  criterion 

Test 

Hiatus  period  (straddled  groups)  and  experience  tracking 

Evaluate  and  Integrate 
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The  paramount  goals  of  a  research  program  would  be  (1)  Facilitating  the  achievement  of  high 
proficiency,  especially  accelerating  across  the  Apprentice  to  Senior  Journeyman  levels  of 
proficiency,  (2)  Facilitating  the  retention  of  expertise  in  the  form  of  both  personnel  capabilities 
and  in  the  form  of  organizational  knowledge,  and  (3)  Producing  applications  for  military 
domains  including  USAF  mission-critical  specializations. 

5.7.6.  Program  Structure  and  Funding 

The  Roadmap  presented  in  Figure  5.2  is  designed  so  as  to  not  be  overly-constraining  and  leave 
open  a  variety  of  possibilities  for  sponsorship,  program  structure,  funding,  and  proposal 
requirements.  For  instance,  a  research  project  on  acceleration  might  be  integrated  with  a  project 
on  retention,  or  the  two  might  be  separate  projects. 

The  following  factors  will  be  formative  of  a  program: 

1) .  Both  research  topics  (acceleration  and  retention)  entail  longitudinal  designs  and  are 
necessarily  mid-term  to  long-term. 

2) .  Both  research  topics  entail  a  requirement  for  a  multi-service  and  multi-disciplinary  research 
team  (see  Section  5.1), 

3) .  Both  research  topics  should  involve  the  study  of  more  than  one  domain;  ideally  domains  that 
are  interestingly  and  substantively  different. 

One  distinct  and  appropriate  program  structure  would  be  the  Multidisciplinary  University 
Research  Initiative  (MURI).  Partners  would  include  university  researchers  with  a  specialization 
in  expertise  studies  and  simulation/scenario  design,  private  sector  partners  with  a  capability  in 
cognitive  systems  engineering,  and  private  sector  partners  with  a  track  record  of  successful 
delivery  of  training  material  and  protocols  to  the  DoD. 

To  be  successful,  such  a  program  would  have  to  be  adequately  funded,  likely  in  the  $10M-i- 
range.  A  promising  course  of  action  would  be  for  the  research  projects  to  select  domains  that  are 
of  interest  and  importance  in  more  than  one  branch  of  the  military  (see  Section  5.3,  above).  In 
this  way,  the  funding  burden  could  be  shared. 

It  is  crucial  for  policy  makers  to  appreciate  the  potential  long-term  benefits  of  successful 
demonstration  of  accelerated  proficiency  and  facilitated  retention.  Consider  the  following: 

•  Suppose  there  is  a  military  job  having  an  existing  cohort  of  100  workers. 

•  Suppose  further  that  in  order  for  each  worker  to  qualify  they  must  go  through  two  years 
of  schoolhouse  training. 

•  Suppose  that  in  order  to  achieve  proficiency  at  the  journeyman  level  (capable  of  solving 
routine  problems  with  no  need  of  supervision  or  assistance)  they  must  have  at  least  3 
years  of  experience  post-schoolhouse,  and  that  in  order  to  achieve  the  junior  expert  level 
(capable  of  innovation  and  adaptation,  and  able  to  solve  rare  and  unusual  cases  on  their 
own)  they  must  have  at  least  5  years  of  experience  post  schoolhouse. 

•  Suppose  that  each  worker  achieves  a  rating  having  an  annual  salary  in  the  range  of 
$75,000  (Paygrades  E-8,  W-3,  0-3;  <20  years  of  service)  to  $95,000  (Paygrades  E-8,  W- 
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3,  0-3;  >20  years  of  service),  there  would  be  a  total  salary  range  of  $7,500,000  to 
$9,000,000. 

•  Suppose  further  that  a  higher  level  of  proficiency  (i.e.,  senior  journeyman)  could  be 
achieved  in  1.5  to  2  fewer  years. 

•  The  net  savings  of  two  years  could  total  $15M  to  $28M. 

Even  assuming  that  the  faster  achievement  of  higher  proficiency  would  lead  to  increased  pay 
grades  over  those  years  of  service,  the  savings  could  potentially  be  enormous,  and  at  the  same 
time  the  acceleration  would  provide  the  military  with  a  workforce  that  is  overall  of  a  higher  level 
of  proficiency. 
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Welcome  to  the  Workshop! 

We  seem  to  have  struck  upon  a  topic  that  is  not  only  important  to  our  nation  but  also  raises 
important  scientific  questions. 

Our  topic  is  the  general  area  of  skill  acquisition  at  the  "high  end"  of  journeymen  and  experts. 
We  define  'skill'  very  broadly  to  include  skills  of  performance,  knowledge,  and  reasoning,  in 
diverse  domains  and  jobs. 

The  scientific  literatures  have  provided  ideas  about  proficiency  scaling.  However,  there  have 
been  no  longitudinal  examinations  that  would  be  of  the  scale  and  scope  required  for  the 
understanding  of  the  development  of  high  levels  of  proficiency. 

The  literature  on  learning  is  vase,  and  includes  many  studies  of  retention,  i.e.,  retention  of  a 
second  language  after  some  period  of  years.  But  we  find  no  studies  examining  retention  of  high- 
end  job  performance  skills  and  knowledge. 

In  the  general  area  of  retention,  little  attention  has  been  given  to  what  happens  when  a 
proficient  worker  has  to  spend  some  period  of  time  on  hiatus. 

Issues  of  long-term  retention  seem  to  be  a  ripe  area  for  scientific  inquiry. 

All  of  these  research  gaps  converge  on  current  national  issues  of  workforce.  In  particular, 
many  knowledge-based  organizations,  including  entire  sectors  of  the  economy,  are  at  risk  due 
to  the  potential  loss  of  expertise. 

Thus,  we  seem  to  have  a  "tiger  by  the  tail." 

In  the  discussions  we  will  have  at  this  Workshop,  we  hope  to  spend  most  of  our  time  in 
collegial  debate  moving  toward  these  goals: 

1 .  Identify  the  most  important  challenge  questions, 

2.  Situate  those  questions  in  the  study  of  domains  that  are  important  to  both  the 
government  and  the  private  sector,  and 

3.  Roadmap  one  or  more  projects  to  study  longitudinal  skill  acquisition  and  retention. 
Thank  you  for  agreeing  to  participate! 


Dr.  Dee  Andrews  Dr.  Robert  Hoffman 


Dr.  Paul  Feltovich 
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Table  Of  Contents 


Welcome  Letter  -  Workshop  Mission  Statement  ? 
What  is  Accelerated  Learning?  Map 
Workshop  Guidelines 
Agenda 

Participant  List 
Presenter  Bios 


Workshop  Guidelines 
Workshop  Rules  of  Order 

<*  Cell  phones  -  please  set  to  OFF  or  vibrate  (and  vibrate,  only  if  the  vibrate  mode  is 
silent). 

We  will  break  for  email  checks  and  phone  checks.  We  understand  that  everyone  is 
busy  and  has  current  obligations  and  responsibilities. 

<*  In  the  group  discussions,  please  have  only  one  conversation  at  a  time. 

<*  Please  be  sure  to  express  all  the  points  that  you  feel  a  need  to  make. 

<*  Please  raise  any  questions  that  you  feel  the  group  might  be  missing. 


Break-Out  Group  Process 

<*  The  goal  of  these  sessions  is  to  capture  challenge  problems  that  will  inform  a  research 
roadmap.  In  order  to  produce  the  best  results,  we  ask  that  you  please  adhere  to  the 
following  instructions: 

<*  Follow  the  guidance  of  your  Group's  facilitator; 

<*  If  your  Group  gets  into  a  mode  of  "idea  generation,”  please  try  to  withhold  criticism 
during  that  idea  generation  phase; 

<*  Please  write  down  every  idea  you  have  on  a  Post-It  Note; 

Please  write  down  every  challenge  question  you  think  of  on  a  Post-It  Note; 

<*  Please  write  large  and  legibly; 

<*  Following  idea  generation,  group  and  filter  your  ideas; 

<*  If  you  have  generated  ideas  for  a  roadmap,  try  to  timeline  it  (near- 
term,  mid-term,  long-term)  and  capture  any  SWOT  that  might  come 
to  mind  (feasibility,  start  date,  duration,  resource  requirements, 
etc...). 
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Agenda 


Friday  October  9,  2009 

8:00  a.m. 

Continental  Breakfast  and  A.M.  Beverage  Service 

8:45  a.m. 

Welcome 

Introductions  and  “Lightning  Round” 

•  Each  participant  conveys  their  focus  area  and  their  key  ideas  or  hot  buttons  on 
issues  of  training,  transfer  and  retention. 

9:45  a.m. 

Presentation  1 

Dr.  Dee  Andrews,  AFRL  Identifying  Interesting  and  Viabie  Research  Chaiienge 
Questions  About  Acceieration  and  Retention 

•  Military  Environment 

•  Long-Term  Retention  Study 

•  Program  Roadmapping 

•  DoD  Accelerated  Learning  Study 

10:15  a.m. 

Full  Group  Discussion 

10:30  a.m. 

Break 

10:45  a.m. 

Presentation  2 

Dr.  Paul  Feltovich  and  Dr.  Robert  Hoffman,  IHMC 

Acceieration  and  Retention:  Research  and  Theory 

1 1 :30  a.m. 

Presentation  3 

Acceleration  and  Retention  Issues 

Lt  Col  Gina  Hilger,  USAF 

Inteiiigence-Surveiiiance,  Reconnaissance  Requirements 

12:15  p.m. 

Lunch  in  the  Garden  Pavilion 

1 :15  p.m. 

Presentation  4 

Acceleration  and  Retention  Issues 

David  Ziebell  (EPRI)  Infrastructure  and  Power  Distribution  Domains 

2:00  p.m. 

Full  Group  Discussion 

2:15  p.m. 

Presentation  5 

Acceleration  and  Retention  Issues 

Col  Daniel  Walker,  USAF-AFRL  UAV Operations 

3:00  p.m. 

Break 

3:15  p.m. 

Break-Out  Groups  (3) 

•  Selection  of  Domains 

4:45  p.m. 

Break 

5:00  p.m. 

Full  Group  Discussion 

•  Research  Design  and  Methodology  Considerations 

•  Closing  comments  and  look  ahead  at  Agenda  for  Day  2 

7:00  p.m. 

Group  Dinner  at  Rancho  de  Tia  Rosa's 
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Agenda 

October  10,  2009 


8:30  a.m. 

Continental  Breakfast  and  A.M.  Beverage  Service 

9:15  a.m. 

(Re)framing  based  on  Day  1  results 

9:30  a.m. 

Presentation  6 

Acceleration  and  Retention  Issues 

Lt  Col  Doyle  Turner,  USAF  Pilot  Transition  Training 

10:00  a.m. 

Break 

10:15  a.m. 

Break-Out  Groups  (3) 

•  Selection  of  Domains 

•  Research  Design  and  Methodology  Considerations 

1 1 :1 5  a.m. 

Break 

1 1 :30  a.m. 

Presentation  7 

Acceleration  and  Retention  Issues 

Dr.  Winston  Bennett,  AFRL  F-16  Retention  Research 

12:15  p.m. 

Lunch 

in  the  Garden  Pavilion 

1 :15  p.m. 

Presentation  8 

Acceleration  and  Retention  Issues 

Dr.  Kevin  Gluck,  AFRL  (with  Dr.  Tiffany  Jaztrzembski) 

Predictive  Performance  Optimizer 

1 :45  p.m. 

Break-Out  Groups  (3) 

•  Challenge  Questions  for  Research 

2:30  p.m. 

Break 

2:45  p.m. 

Fuii  Group  Brainstorm 

•  From  Challenge  Questions  to  a  Research  Roadmap 

3:45  p.m. 

Break 

4:00  p.m. 

Fuii  Group  Brainstorm  continued 

•  From  Challenge  Questions  to  a  Research  Roadmap 

5:00  p.m. 

Ciosing  Discussion:  Lookahead 
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247 


Participant  List 


Dee 

Andrews 

Air  Force  Research  Laboratory  -  Mesa 

dee.andrews@mesa.afmc.af.mil 

Herb 

Bell 

Air  Force  Research  Laboratory  -  Mesa 

herbert.bell@mesa.afmc.af.mil 

Wink 

Bennett 

Air  Force  Research  Laboratory  -  Mesa 

Winston. bennett@mesa.afmc.af. 
mil 

Jeff 

Bradshaw 

Institute  for  Human  &  Machine  Cognition 

jbradshaw@ihmc.us 

Neil 

Charness 

Florida  State  University  -  Dept  of  Psychology 

charness@psy.fsu.edu 

Lia 

DiBello 

WTR  Inc. 

lia@wtri.com 

Paul 

Feltovich 

Institute  for  Human  &  Machine  Cognition 

pfeltovich@ihmc.us 

Steve 

Fiore 

Institute  for  SIM  &  Training 

sfiore@ist.ucf.edu 

Pat 

Fitzgerald 

Air  Force  Research  Laboratory  -  Mesa 

patricia.fitzgerald@mesa.afmc. a 
f.mil 

Jared 

Freeman 

Aptima,  Inc. 

freeman@aptima.com 

Andy 

Gibbons 

Brigham  Young  University 

andy gibbons@byu.edu 

Kevin 

Gluck 

Air  Force  Research  Laboratory  -  Mesa 

kevin.gluck@mesa.afmc.af.mil 

Steve 

Goldberg 

Army  Research  Institute 

stephen.goldberg@us.army.mil 

Art 

Graesser 

University  of  Memphis 

a.graesser@mail.psyc.memphis. 

edu 

Frank 

Greitzer 

Pacific  Northwest  National  Laboratory 

frank.greitzer@pnl.gov 

Alice 

Healy 

University  of  Colorado-Dept  of  Psychology 

ahealy@psych.colorado.edu 

Lt.  Col.  Gina 

Hilger 

United  States  Air  Force  -  Shaw  AFB 

gina.hilger@shaw.af.mil 

Robert 

Hoffman 

Institute  for  Human  &  Machine  Cognition 

rhoffman@ihmc.us 

Robert 

Hubal 

RTI  International 

rhubal@rti.org 

Gary 

Klein 

Klein  Associates 

gklein@ara.com 

Alex 

Lincoln 

The  Andrews  Institute 

alex.lincoln@theandrewsinstitut 

e.com 

Lochlan 

Magee 

DRDC-  Collaborative  Performance  and 
Learning  Section 

lochlan.  magee@drdc-rddc.gc.ca 

Harry 

O'Neil 

University  of  Southern  California 

hooneil@cse.ucla.edu 

Robert 

Patterson 

Washington  State  University 

rpatter@mail.wsu.edu 

Byron 

Pierce 

Air  Force  Research  Laboratory  -  Mesa 

byron.pierce@mesa.afmc.af.mil 

Robert 

Proctor 

Purdue  University 

proctor@psych.purdue.edu 

Rand 

Spiro 

Michigan  State  University 

rspiro@msu.edu 

Lt  Col  Doyle 

Turner 

USAF  -  Luke  AFB 

doyle.turner@luke.af.mil 

Kurt 

VanLehn 

Arizona  State  University 

kurt.vanlehn@asu.edu 

Karen 

Velkey 

North  rup-Grumman 

karen.velkey@ngc.com 

Col  Daniel 

Walker 

Air  Force  Research  Laboratory  -  Mesa 

daniel.walker@mesa.afmc.af.mil 

David 

Ziebell 

Electric  Power  Research  Institute 

dziebell@epri.com 

248 


Appendix  B 

White  Paper  Read-ahead 


249 


About  the  Meeting 

"Accelerated  Learning  and  Facilitated  Retention  oe  Proeiciency" 
Robert  R.  Hoffman,  Paul  Feltovich,  Stephen  M.  Fiore,  Gary  Klein,  Dee  Andrews 


Introduction 

The  objeetive  of  this  effort  is  to  ereate  a  design  and  roadmap  for  investigations  aimed  at 
developing  robust  and  broadly-applieable  methods  for: 

(1)  Aeeelerating  the  aehievement  of  profieieney  in  USAF  mission-eritieal  speeializations  (and 

analogous  domains  in  the  private  seetor),  and 

(2)  Facilitating  the  retention  of  knowledge  and  skill,  especially  for  military  personnel  who  must 

take  long  hiatuses  from  their  domain  of  primary  expertise  in  order  to  fulfill  other  military 

requirements. 

USAF  personnel,  especially  officers,  are  required  to  leave  those  domains  to  perform  duties  such 
as  staff  duty,  teaching  at  a  military  academy  or  in  Reserve  Officer  Training  Corps  programs,  and 
taking  courses  of  study  at  military  or  civilian  institutions  of  higher  education.  These  hiatuses  can 
last  from  a  few  weeks  to  as  long  as  four  years.  During  these  hiatuses  it  often  not  possible  to  train 
or  practice  in  their  domain  of  primary  expertise. 

Proficiency  is  critical  to  performance  in  complex  work  contexts,  including  the 
transformed  strategic  environment  that  confronts  the  USAF.  During  the  Cold  War,  USAF 
personnel  could  typically  count  on  staying  in  one  place  for  assignment  for  at  least  three  years 
before  rotating.  That  era  allowed  for  robust  continuous  training  while  at  a  duty  station.  However, 
in  this  current  era  of  frequent  deployments  to  a  variety  of  locations  worldwide  to  fight  the  War 
on  Terror,  there  are  far  fewer  opportunities  to  have  systematic  training  and  practice.  Yet,  it  still 
ordinarily  takes  many  years  to  achieve  proficiency.  Therefore,  there  would  be  great  advantages  if 
the  USAF  could  establish  regimens  of  training  that  could  accelerate  the  achievement  of  that 
proficiency.  There  is  reason  to  hypothesize  that  the  achievement  of  proficiency  can  be 
accelerated,  and  the  proposed  working  meeting  will  generate  specific  plans  and  roadmaps  for 
actual  attempts  to  do  so.  It  is  likely  that  methods  for  acceleration  will  leverage  the  technologies 
and  capabilities  including  virtual  training,  cross  training,  training  across  strategic  and  tactical 
levels,  and  training  for  resilience  and  adaptivity. 

The  proposed  working  meeting  would  enable  and  support  a  larger-scale  effort  to 
culminate  in  a  robust  and  broadly-pplicable  suite  of  methods  for  facilitating  the  achievement  and 
retention  of  proficiency. 


Background 

A  workshop  sponsored  by  the  Office  of  the  Director,  Defense  Research  and  Engineering 
and  the  DoD  Accelerated  Learning  (AL)  Technology  Focus  Team  (July  2008)  brought  together 
leading  academic,  private  sector,  and  DoD  specialists  in  areas  of  training  and  expertise  studies. 
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to  discuss  the  goal  of  accelerated  learning.  Presentations  at  that  workshop  laid  out  the  major 
challenges  and  issues  for  accelerated  learning,  and  the  activities  proposed  here  will  extend  and 
refine  the  results  of  that  effort,  along  with  an  initial  "road-map"  for  investigations  in  mission- 
critical  areas  of  the  Air  Force.  In  addition,  the  challenge  of  competency  retention  for  personnel 
taking  hiatuses  will  be  explored. 

While  many  military  jobs  can  be  trained  through  established  COTS  methods,  and  those 
tasks  can  be  performed  by  journeymen,  the  military  needs  highly  proficient  personnel  who  can 
perform  tasks.  The  bottleneck  for  producing  such  critical  individuals  is  that  it  has  typically  taken 
many  years  of  experience  and  deliberate  practice  for  individuals  to  master  their  domains  (e.g., 
anti-submarine  warfare;  see  Ericsson  et  ah,  2006).  Reasons  for  this  include  domain  complexity, 
irregularity  across  encountered  cases,  the  need  for  deliberate  practice,  and  the  need  for  practice  at 
tough  cases  to  stretch  skill. 

Domain  specialists  provide  technical  judgment  to  speed  decision-making  in  time-critical 
events.  They  provide  resilience  to  operations  by  resolving  tough  problems,  anticipating  future 
demands  and  re-planning,  and  acting  prudently  by  judgment  rather  than  by  rule.  Specialists 
exercise  effective  technical  leadership  in  ambiguous  or  complex  situations,  often  by 
communicating  subtle  features  that  other  people  will  not  see  until  they  are  pointed  out  (Hoffman, 
1997).  Often  they  are  also  the  ones  who  understand  the  history,  the  interdependencies  of  units 
and  processes,  and  the  culture  of  their  complex  organizations,  knowledge  that  is  often  essential 
in  actually  "getting  things  done"  (e.g.,  Stein,  1997). 

The  challenge  of  learning  is  compounded  in  the  military  by  such  practices  as  collateral 
assignment,  frequent  redeployment  (e.g.,  rapid  skill  decay  on  the  part  of  pilots  creates  a  need  for 
expensive  re-training),  inadequate  or  ad  hoc  mentoring,  and  the  drive  for  “just-in-time”  training. 
Another  significant  challenge  is  clustered  around  career  (versus  job)  training,  and  expertise 
retention.  Professional  Military  Education  offered  by  such  intuitions  as  the  various  war  colleges 
is  an  example  of  career  training.  There  personnel  learn  about  operational  and  strategic 
warfighting  issues.  Indeed,  the  entire  field  of  “knowledge  management”  is  formed  around  the 
notion  of  preserving  and  sharing  expertise  (e.g.,  Becerra-Eemandez,  et  ah,  2004). 

Transfer,  or  the  ability  to  use  knowledge  flexibly  and  effectively  across  application  areas, 
is  an  important  component  of  proficiency.  In  large  respect,  accelerated  learning  means  improving 
transfer  (and  retention)  capability.  The  major  theory  of  transfer  in  learning  is  the  "common 
elements"  theory.  Based  on  this  idea,  one  would  say  that  training  should  minimize  the  transfer 
distance  from  training  to  workplace.  Recent  research  is  suggestive  of  the  conditions  that  promote 
transfer,  such  as  the  judicious  use  of  particular  kinds  of  feedback  (Schmidt  and  Bjork,  1992).  Eor 
instance,  summary  feedback  is  less  conducive  of  initial  learning  than  feedback  after  each  trial, 
but  summary  feedback  during  learning  does  seem  to  promote  delayed  retention. 

Performance  issues  go  beyond  transfer  from  the  classroom  to  the  operational  context. 
Simply  “working  at  a  job”  does  not  promote  progression  along  the  proficiency  continuum  (e.g., 
Eeltovich,  Prietula,  and  Ericsson,  2006).  Unless  there  is  continuous  deliberate  practice  and 
feedback  at  difficult  tasks,  the  only  thing  one  can  do  “on  the  job”  is  forget  and  actually 
experience  degradation  of  skill. 
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Furthermore,  the  current  challenges  for  military  training  involve  two  different  sorts  of 
transfer.  One  is  transfer  across  mission  types.  An  example  would  be  an  infantry  commander, 
who  knows  traditional  warfare,  who  is  asked  to  develop  tactics  for  an  insurgency  operation.  The 
second  challenge  is  transfer  across  responsibilities.  An  example  would  be  a  warfighter  having  a 
skill  at  maintenance  of  an  F-16  engine  who  is  promoted  to  a  supervisory  position.  Since  different 
skill  sets  would  be  involved,  one  would  need  to  train  for  the  new  role,  and  not  just  assume 
transfer  would  or  might  happen.  When  a  journeyman  or  expert  moves  from  a  job  where  domain 
expertise  is  all  that  is  required  of  them  to  a  supervisory  job,  there  is  extra  challenge  because  they 
are  still  expected  to  maintain  their  domain  competence  while  also  acquiring  and  performing  their 
new  supervisory  skills. 

The  2008  Workshop  successfully  pointed  to  capability  gaps.  The  challenge  now  is  how  to 
a  structure  research  questions  and  a  research  strategy  to  help  the  military  address  the  gaps. 
Suggestions  for  acceleration  methods  include  leveraging: 

•  The  important  features  of  deliberate  practice  and  mentoring, 

•  The  technologies  of  virtual  reality  training, 

•  Multi-media  for  training  at  complex  tasks, 

•  Computer  technology  to  develop  and  manage  libraries  of  representative  and  tough  tasks, 

•  Computer  technology  to  develop  and  manage  knowledge  bases  that  capture  expert 
knowledge  and  skill,  and 

•  New  cognitively  based  methods  for  designing  instruction. 

The  technical  research  community  is  now  positioned  to  work  specifically  on  the  question 
of  how  to  create  robust  methods  to  accelerate  the  achievement  of  proficiency  and  facilitate  its 
retention.  Such  objectives  are  the  focus  for  a  follow-on  Working  Meeting  on  Accelerated 
Learning  and  retention  of  competency  that  is  the  core  component  of  the  project  we  are 
proposing. 

Retention  Issues 

An  important  part  of  the  workshop  is  devoted  to  the  reacquisition  and  retention  of  competence 
we  relate  below  some  key  principles  of  learning  retention.  These  principles  from  the  literature 
are  key  for  understanding  retention  for  short  term  hiatuses  from  the  domain  of  interest  (e.g., 
weeks  or  months).  However,  there  is  very  little  literature  and  even  less  data  about  the  types  of 
long  term  hiatuses  (up  to  four  years)  that  many  military  personnel  face.  How  to  research  such 
longitudinal  situations  will  be  a  major  focus  of  the  workshop. 

Active  Versus  Passive  Learning 

Retention  of  knowledge  and  skill  is  better  when  material  at  the  time  of  acquisition  is  processed 
deeply,  embellished,  and  connected  to  and  integrated  with  other  knowledge  -  in  general,  when 
knowledge  is  not  compartmentalized,  but  is  richly  structured  and  indexed.  Material  to  be  learned 
is  actively  manipulated,  rather  than  being  merely  repeated  in  learning.  Manipulations  supporting 
active  learning  may  include  extrapolating,  discussing.  Concept  Mapping,  alternative  perspective 
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taking,  practicing  on  related  problems  and  cases,  creating  related  projects,  and  so  forth 
(Bransford,  Brown  and  Cocking,  2000;  see  also  Canas,  et  ak,  2003). 

Tapping  the  Deep  Structure  and  Developing  Rich  Understanding 

Retention  of  knowledge  and  skill  is  better  when  complex  material  is  “understood,”  rather  than 
learned  by  rote;  when  concepts  principles,  and  rules  complement  or  supplement  teaching  of  rote 
knowledge  or  facts.  Novices  approach  complex  problems  much  differently  from  experts. 
Novices  code  problems  in  term  of  their  surface  features  (e.g.,  the  objects  involved)  and  try  to 
link  (meager)  solutions  to  these  interpretations,  which  in  most  complex  fields  cannot  support  rich 
problem  solving.  Experts,  in  turn,  understand  problems,  by  encoding  their  "deep"  structure, 
often  functional,  integrated  relationships  that  can  be  functionally  tied  to  solutions  (Chi,  Feltovich 
and  Glaser,  1981).  These  deep  structures  are  abstract,  and  hence  support  both  transfer  and 
retention  (Zeitz,  1997). 

Redundant  Indexing,  Multiple  Representations 

A  fact  of  our  modern  world  is  change  and  diversity.  People  are  expected  to  have  multiple  jobs 
over  a  career,  jobs  themselves  change,  customization  is  required,  work  tools  change,  technology 
encroaches,  and  so  forth.  Knowledge  and  skills  that  will  need  to  be  used  in  many  ways  must  be 
practiced  in  many  ways,  construed  in  many  ways,  connected  and  indexed  in  many  ways,  and 
represented  in  many  ways  (e.g.,  Spiro,  et  ah,  1992,  Spiro,  Thota,  Collins  and  Feltovich,  2004). 
This  affects  retention  by  supporting  redundant  indexing,  cross-case  connectivity,  and  cognitive 
flexibility  in  dealing  with  novel  instances  over  time.  (Spiro,  et  ah,  1988). 

Understanding  Concepts  in  Context  and  Making  Fine  Discriminations 

Experts  develop  vast  memory,  organized  for  finely  tuned  cases  and  case-elements  within  their 
field  of  mastery  (Ericsson  et.  ah,  2006).  This  helps  them  finely  discriminate  case  elements  as 
indexes  for  just  the  right  structure(s)  in  memory.  Well-tuned  case  structure  memories  enable 
better  indexing,  anticipation  of  "what's  next,"  and  understanding  of  when  an  initially  triggered 
model  for  interpretation  is  ill-suited  (i.e.,  has  gone  down  the  wrong  path)  (Feltovich,  Spiro,  & 
Coulson,  1997).  This  affects  retention  in  that  it  is  important  that  the  right  information  is  recalled 
flexibly  over  time. 

Cognitive  Transformation  and  Undoing  Bad  Learning 

Unfortunately,  it  has  been  found  that  in  areas  of  learning  that  are  difficult,  learners  can  develop 
highly  engrained,  overly  simplistic  misconceptions  that  can  impede  greater  understanding  over 
time.  This  affects  retention  in  that,  even  with  long-term  practice,  it  is  wrong  understandings  that 
are  being  reinforced,  maintained.  Methods  of  instruction  based  on  Cognitive  Transformation 
Theory  (Klein  and  Baxter,  2009),  and  Cognitive  Flexibility  Theory  (Spiro,  et  ah,  1988)  are 
aimed  at  undermining  faulty  conceptions  (as  well  as  supporting  the  development  of  better  ones). 

Practice  and  Continuous  Learning 

Clearly,  the  best  means  for  retaining  knowledge  and  skills  is  to  never  stop  practicing  and 
learning.  This  is  the  bedrock  of  expertise  research,  reflected  in  the  concept  of  life-long 
"deliberate  practice"  (Ericsson  et  ah,  2006).  It  has  even  been  shown  that,  for  experts,  knowledge 
and  skills  directly  from  their  field  of  expertise  are  the  last  to  degrade  with  the  advent  of  age- 
related  dementia,  perhaps  the  ultimate  retention  (e.g.,  Krampe  and  Charness,  2006). 
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Transfer-Appropriate  Processing 

Retention  is  more  effective  when  conditions  of  training  and  learning  resemble  those  in  which  the 
learned  knowledge  and  skill  will  be  applied.  This  condition  for  retention  and  transfer  is 
complicated  in  ill- structured  domains,  where  there  can  be  special  demands  for  knowledge 
transfer  that  result  from  the  likely  variability  between  conditions  of  initial  learning  and  later  use. 
This  accentuates  the  need  for  training  that  uses  multiple  cases,  multiple  perspectives,  and 
multiple  goals  for  knowledge  use  in  the  course  of  learning. 


Focal  Points  for  the  Working  Meeting 

The  following  core  topics  will  be  addressed. 

Measures  and  Levels  of  Proficiency 

This  topic  addresses  the  measurement  methodology  to  determine  levels  of  proficiency.  This 
needs  to  go  beyond  the  traditional  rough  cuts  often  used  in  organizations  and  expertise  studies 
(i.e.,  expert  versus  novice)  to  achieve  finer  discriminations  along  the  skill  continuum  (e.g., 
Hoffman,  1998).  The  challenge  is  to  establish  reliable  and  valid  markers  that  might  be  used  for 
such  things  as  placement,  promotions,  and  for  use  in  assigning  practitioners  to  skill  levels  in 
research  studies.  In  this  regard,  measurement  of  performance  for  all  complex  sociotechnical 
systems  must  also  go  well  beyond  traditional  measures  of  the  performance  of  individuals  (hits, 
errors,  rate,  etc.)  (Alberts  and  Hayes,  2003). 

Selection  and  Scaffolding  for  Development 

Traditional  hiring  practices  have  not  served  well  to  predict  who  will  eventually  be  a  high 
performer.  New  methods  need  to  be  investigated.  For  instance,  it  may  be  that  common  screening 
tools  at  hiring  will  need  to  be  replaced  or  augmented  by  various  indicators  that  only  become 
apparent  after  some  period  of  time  functioning  on  the  job.  Along  with  identification,  there  will 
need  to  be  programs  for  supporting  progressive  development  beyond  what  are  typically  taken  to 
be  "professional  development"  activities  in  many  organizations. 

Mentoring  and  Apprenticeship 

Many  aspects  of  work  are  best  learned  "at  the  bench,"  under  guidance  from  a  skilled  mentor. 
These  include  what  is  often  called  "tacit  knowledge,"  that  is,  many  aspects  of  "know  how," 
where  opportunities  for  learning  largely  manifest  themselves  in  the  course  of  real,  active  work,  in 
the  social  context  of  work  (Schon,  1983).  We  need  to  be  able  to  identify  excellent  mentors,  study 
them  to  reveal  the  knowledge  and  strategies  of  mentors,  and  develop  programs  for  training 
mentors.  We  also  need  to  determine  what  job  practices  are  best  suited  for  mentoring,  as  opposed 
to  using  other  kinds  of  training  techniques. 

Practice  and  Experience  Compression 

It  is  well  known  that  extensive  experience  with  challenging  cases  and  deliberate  practice  (and 
guidance)  are  hallmarks  of  the  development  of  expertise.  One  likely  reason  for  the  "10,000 
hours"  guideline  for  the  development  of  expertise  is  that  ordinary  work  flow  does  not  provide 
enough  exposure  to  "tough  cases,"  those  that  stretch  skill,  and  does  not  present  them  within  any 
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kind  of  logical  structure  for  learning.  Finding  ways  to  eompaet  experiences  will  be  vital  to  any 
program  aiming  to  accelerate  expertise.  Methods  might  include  games,  simulations,  virtual 
worlds,  decision-making  exercises,  and  other  kinds  of  case-oriented  delivery  systems  (e.g., 
Spiro,  Thota,  and  Feltovieh,  2003). 

Scaling  Cases 

Experience  with  cases  is  eritical  to  the  development  of  expertise,  as  is  compression  of  these 
experiences.  Researeh  to  reliably  test  degree  of  learning,  retention,  and  adaptation,  will  need  a 
set  of  dependable  seenarios  or  cases  that  can  be  scaled  for  difficulty,  and  that  span  a  range  of 
situations.  Well-known  knowledge  elicitation  methods  (e.g.,  the  Critical  Decision  Method; 
Crandall,  Klein  and  Hoffman,  2006)  can  be  employed,  with  participants  who  span  the 
profieiency  seale  (i.e.,  novice,  apprentice,  journeyman,  expert)  to  elieit  cases.  Henee,  it  will  be 
necessary  to  develop  a  large  ease  repository  in  which  cases  are  sealed  for  difficulty  relative  to  the 
different  levels  of  profieiency.  For  each  level  we  will  need  cases  that  are  fairly  ordinary  at  that 
level,  but  also  "level  stretchers" —  tough  cases  that  push  capability  beyond  the  current  rank. 
Such  a  case  library  can  be  used  in  training,  evaluation,  and  researeh  projects. 

Designing  Case  Libraries 

Case  experienee  is  so  important  to  expertise  that  it  ean  be  envisioned  that  organizations  will 
develop  very  large  case  repositories  for  use  in  training,  but  also  to  preserve  organizational 
memory.  Instruetion  using  cases  is  greatly  enhanced  when  "just  the  right  case"  or  set  of  cases  ean 
be  engaged  at  a  prime  learning  moment  for  a  learner  (Kolodner,  1993).  (This  also  argues  for 
large  numbers  of  cases,  to  eover  many  eontingeneies)  This  is  a  matter  of  organization  among 
eases,  good  retrieval  schemes,  and  smart  indexing.  Note  that  an  added  benefit  from  the  creation 
of  case  repositories  is  that  the  process  of  developing  cases  is  itself  a  form  of  knowledge 
elicitation,  and  therefore  it  feeds  direetly  into  the  proeess  of  capturing  expert  knowledge. 

Learning  for  Flexibility,  Adaptability,  and  Resilience 

Robustness  is  the  ability  to  maintain  effectiveness  across  a  range  of  tasks,  situations,  and 
conditions.  Resilience  is  the  ability  to  reeover  from  a  destabilizing  perturbation  in  the  work  as  it 
attempts  to  reaeh  its  primary  goals.  Adaptivity  is  the  ability  to  employ  multiple  ways  to  suceeed 
and  the  capacity  to  move  seamlessly  among  them.  Fundamental  to  the  achievement  of 
robustness,  resilience  and  adaptivity  is  the  opportunity  to  practice  at  problems  that  stretch  current 
eompeteney  (Ericsson  and  Eehmann,  1996;  Eeltovich,  et  ah,  1997).  However,  modern  work 
presents  ehallenges  for  achieving  sueh  adaptive  skills.  Eor  mueh  of  history,  individuals  have 
developed  mastery  and  resilience  by  foeusing  intensely  on  a  relatively  circumscribed  task  or 
suite  of  skills,  e.g.,  chess  or  welding.  Modern  work  (espeeially  in  socioteehnical  work  systems) 
has  beeome  mueh  more  unstable  and  eomplex.  People  change  jobs,  jobs  ehange,  orders  ehange, 
teehnology  intrudes  and  ehanges  the  work,  eustomization  in  products  and  services  requires 
versatility  beyond  the  assembly  line,  etc.  Can  we  train  (or  select  for)  expertise  in  flexibility  and 
resourcefulness?  What  kind(s)  of  learning  and  materials  does  it  require?  Eor  example,  one 
method  that  has  been  used  is  to  have  workers  rotate  through  other  job  roles  that  are  integrated 
with  their  own  work. 

Addressing  All  of  These  Factors,  and  More,  in  the  Context  of  Teams  and  Groups 
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In  addition  to  fluidity,  customization,  etc.,  another  prominent  feature  of  modern  work  involves 
teams  and  teamwork.  For  functional,  workable  acceleration  to  be  possible,  training  exercises 
must  encourage  the  acquisition  of  teamwork  skills.  An  expert  group  is  not  the  same  as  a  group  of 
experts  (Salas,  et  ah,  2006).  While  many  skills  may  be  in  common,  teamwork  engages  others,  for 
example,  certain  kinds  of  "people  skills,"  coordination  skills,  understanding  of  others  and  their 
"ways,"  and  the  ability  to  develop  and  maintain  satisfactory  mutual  understanding,  "common 
ground"  (Klein,  Feltovich,  Bradshaw,  and  Woods,  2004). 

Reacquisition  of  Competency 

As  discussed  above,  military  personnel  face  challenges  of  not  only  retaining  competency,  but 
also  reacquiring  competency  once  it  has  been  degraded  through  non-use  after  long  hiatuses.  This 
phenomenon  caused  by  perhaps  years  away  from  domains  of  primary  competency  is  relatively 
unique  to  the  military  and  has  not  really  been  studied  by  learning  researchers.  Therefore,  we  are 
left  to  largely  wonder  if  principles  of  reacquisition  that  have  been  studied  in  shorter  hiatuses 
might  be  applicable  to  the  situation  faced  in  the  military,  or  will  empirical  study  reveal  new 
principles. 


Specific  Meeting  Activities 

The  Working  Meeting  will  have  a  focus  on  identifying  important  and  difficult  areas  of  Air  Force 
and  civilian  work-practice  for  which  it  is  feasible  to  conduct  actual  programs  of  research  to 
accelerate  and  retain  expertise,  and  to  reacquire  expertise  once  degraded  or  lost  completely. 

The  Working  Meeting  will  have  a  goal  of  generating  detailed  roadmaps  for  how  one  or  more 
such  research  projects  would  be  planned  and  conducted,  including  the  resources  and  capabilities 
required. 

Identify  candidate  sub-domains  for  analysis  and  focus:  We  will  begin  to  identify  domains  that 
might  be  a  focus  for  attempts  to  accelerate  the  achievement  of  expertise.  We  will  seek  mission 
critical  areas  that  manifest  some  combination  of  these  features: 

•  Hard  to  learn  and  do  well,  as  attested  to,  for  instance,  by  high  wash-out  rates  and  few 
extant  experts; 

•  Have  in  the  past  required  very  long  training  and  on-the-job  experience; 

•  Demonstrate  complexity  in  that  they  involve  many  interacting  components  and 
stakeholders;  are  dynamic; 

•  Are  irregular  in  that  different  versions  of  the  "same  problem"  may  look  very  different; 
and  Reflect  complex,  dynamic  and  nonlinear  patterns  of  causation,  and  decision  making 
under  uncertainty. 

•  Have  potential  for  significant  cross-over  applications  of  accelerated  learning  to  the 
civilian  sector  and  are  also  possible  candidate  domains  for  analysis,  such  as  weather 
forecasting. 

The  road-mapping  effort  will  converge  on  the  identification  of  at  least  two  (and  ideally 
three)  domains  that  are  interestingly  as  well  as  substantively  different,  that  are  mission  critical, 
and  that  depend  on  expertise  and  high  levels  of  proficiency  (see  the  discussion  above  on  the 
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identification  of  candidate  sub-domains.  While  one  might  seek  to  design  and  conduct  a  single 
study  on  acceleration,  in  some  single  selected  domain,  that  would  serve  only  as  a  demonstration 
and  would  not  capitalize  on  the  potential  of  the  proposed  working  meeting.  Thus,  the  anticipated 
product  would  include  more  than  one  roadmap  for  acceleration  projects  in  more  than  one 
domain,  to  allow  scientific  comparisons  and  convergence. 

References 

Alberts,  D.  S.  and  Hayes,  R.  E.  (2003).  Power  to  the  edge:  Command  and  control  in  the  information  age.  C2 
Research  Program  Publications. 

Becerra-Femandez,  L,  Gonzalez,  A.,  and  Sabherwal,  R.,  (2004).  Knowledge  management:  Challenges,  solutions, 
and  technologies.  Englewood  Cliffs,  NJ;  Prentice  Hall. 

Bransford,  J.D.,  Brown,  A.L.  and  Cocking  (2000).  How  people  learn.  Washington,  DC;  National  Academy  of 
Sciences. 

Canas,  A.  J.,  Coffey,  J.  W.,  Carnot,  M.  J.,  Feltovich,  P.  J.,  Hoffman,  R.  R.,  Feltovich,  J.  and  Novak,  J.  D.  (2003).  A 
Summary  of  Literature  Pertaining  to  the  Use  of  Concept  Mapping  Techniques  and  Technologies  for  Education  and 
Performance  Support."  Report  to  the  Chief  of  Naval  Education  and  Training,  Pensacola  EL,  from  the  Institute  for 
Homan  and  Machine  Cognition,  Pensacola,  EL. 

Chi,  M.,  Feltovich,  P.  and  Glaser,  R.  (1981).  Categorization  and  representation  of  physics  problems  by  experts  and 
novices.  Cognitive  Science,  5,  121-152. 

Crandall,  B.,  Klein,  G.,  and  Hoffman  R.  R.  (2006).  Working  Minds:  A  Practitioner’s  Guide  to  Cognitive  Task 
Analysis.  Cambridge,  MA;  MIT  Press. 

Ericsson,  K.  A.,  Charness,  N.,  Feltovich,  P.  J.,  and  Hoffman,  R.  R.,  (Eds.)  (2006).  Cambridge  handbook  of  expertise 
and  expert  performance.  New  York;  Cambridge  University  Press. 

Ericsson,  K.  A.  and  Lehmann,  A.  C.  (1996).  Expert  and  exceptional  performance;  Evidence  on  maximal  adaptations 
on  task  constraints.  Annual  Review  of  Psychology,  47,  273-305. 

Farr,  M.  J.  (1987).  The  long-term  retention  of  knowledge  and  skills.  New  York;  Springer-Verlag. 

Feltovich,  P.  J.,  Ford,  K.  A.,  and  Hoffman,  R.  R.  (Eds.)  (1997).  Expertise  in  context.  Cambridge,  MA;  The  MIT 
Press. 

Feltovich,  P.J.,  Prietula,  M.J.,  and  Ericsson,  K.A  (2006).  Studies  of  expertise  from  psychological  perspectives.  In 
K.A.,  Ericsson,  N.  Charness,  P.J.  Feltovich,  and  R.  R.  Hoffman  (Eds.),  Cambridge  handbook  of  expertiseand  expert 
performance  (pp.  41-67).  Cambridge,  UK;  Cambridge  University  Press. 

Feltovich,  P.  J.,  Spiro,  R.J.  and  Coulson,  R.L  (1993).  Learning,  teaching  and  testing  for  complex  conceptual 
understanding.  In  N.  Frederiksen,  R.  Mislevy,  and  I.  Bejar  (Eds.),  Test  theory  for  a  new  generation  of  tests. 
Hillsdale,  NJ;  Lawrence  Erlbaum  Associates. 

Feltovich,  P.  J.,  Spiro,  R.  J.  and  Coulson,  R.  L  (1997).  Issues  of  expert  flexibility  in  contexts  characterized  by 
complexity  and  change.  In  P.  J.  Feltovich,  K.  M.  Ford,  and  R.  R.  Hoffman  (Eds.),  Expertise  in  context:  Human  and 
machine,  (pp.  125-146),  Cambridge;  AAAI/MIT  Press. 

Hoffman,  R.  R.  (Ed.)  (1992).  The  psychology  of  expertise:  Cognitive  research  and  empirical  AT  Mahwah,  NJ; 
Erlbaum. 


257 


Hoffman,  R.  R.  (1998).  How  can  expertise  be  defined?;  Implications  of  research  from  cognitive  psychology.  In  R. 
Williams,  W.  Faulkner  and  J.  Fleck  (Eds.),  Exploring  expertise  (pp.  81-100).  New  York;  Macmillan. 

Hoffman,  R.  R.  (Ed.)  (2007).  Expertise  out  of  context:  Proceedings  of  the  Sixth  International  Conference  on 
Naturalistic  Decision  Making.  Boca  Raton  EL;  Taylor  and  Erancis. 

Hoffman,  R.  R.,  Eiore,  S.  M.,  Klein,  G.  and  Feltovich,  P.  (July,  2008).  Accelerating  the  Achievement  of  Mission- 
Critical  Expertise.  Report  to  the  Electric  Power  Research  Institute,  Palo  Alto,  CA. 

Klein,  G.  and  Baxter,  H.C.  (2009).  Cognitive  transformation  theory;  Contrasting  cognitive  and  behavioral  learning. 
In  The  PSI  handbook  of  virtual  environments  for  training  and  education:  Developments  for  the  military  and 
beyond,  Vol.l,  Education:  Learning,  requirements  and  metrics  (pp.  50-65).  Praeger  Security  International. 

Klein,  G.,  Feltovich,  P.J.,  Bradshaw,  J.M.,  and  Woods,  D.D.  (2005).  Common  ground  and  Communication  in  joint 
activity.  In  W.B.  Rouse  and  K.R.  Boff,  (9rga«/zflrio«a(  (pp.  139-184).  Hoboken,  NJ;  Wiley. 

Kolodner,  J.L  (1993).  Case-based  reasoning.  San  Meteo,  CA;  Morgan-Kaufmann. 

Krampe,  R.  T.  and  Charness,  N.  (2006).  Aging  and  expertise.  In  K.  A.  Ericsson,  N.  Charness,  P.  J.  Eeltovich  and  R. 
R  Hoffman,  (Eds.)  Cambridge  handbook  of  expertise  and  expert  performance  (pp.  723-742).  New  York;  Cambridge 
University  Press. 

Salas,  E.,  Rosen,  M.A.,  Burke,  C.S.,  Goodwin,  G.E.,  and  Eiore,  S.M.  (2006).  The  making  of  a  dream  team;  What 
expert  teams  do  best.  In  K.A.  Ericsson,  N.  Charness,  P.J.  Eeltovich,  and  R.  R.  Hoffman  (Eds.),  Cambridge  handbook 
of  expertise  and  expert  performance  (pp. 439-453).  Cambridge,  UK;  Cambridge  University  Press. 

Schmidt,  R.  A.  and  Bjork,  R.  A.  (1992).  New  conceptualizations  of  practice;  Common  principles  in  three  paradigms 
suggest  new  concepts  for  training.  Psychological  Science,  3,  207-217. 

Schon,  D.A  (1983).  The  reflective  practitioner.  New  York;  Basic  Books. 

Spiro,  R.J.,  Collins,  B.P.,  Thota,  J.T.,  and  Eeltovich,  P.J.  (2003).  Cognitive  flexibility  theory;  Hypermedia  for 
complex  learning,  adaptive  knowledge  application,  and  experience  acceleration.  Educational  Technology. 

Spiro,  R.,  Coulson,  R.,  Eeltovich,  P.,  and  Anderson,  D.  (1988).  Cognitive  flexibility  theory;  Advanced  knowledge 
acquisition  in  ill-structured  domains.  In  Proceedings  of  the  10'''  Annual  Conference  of  the  Cognitive  Science  Society. 
Hillsdale,  NJ;  Lawrence  Erlbaum  Associates. 

Zeitz,  C.  M.  (1997).  Some  concrete  advantages  of  abstraction;  How  experts'  representations  facilitate  reasoning.  In 
P.  J  Eeltovich,  K.  M.  Lord  and  R.  R.  Hoffman  (Eds.).  Expertise  in  context  (pp.  43-65).  Menlo  Park,  CA;  MIT/AAAI 
Press. 


258 


Appendix  C 


Biographical  sketches  oe  the  working  Meeting  Participants 


259 


Dee  H.  Andrews 
U.S.  Air  Force 


Dr.  Dee  H.  Andrews  is  a  Senior  Scientist  (ST)  with  the  Human  Effectiveness  Directorate,  711* 
Human  Performance  Wing,  of  the  Air  Force  Research  Laboratory  in  Mesa,  Arizona.  Previously 
he  held  the  position  of  Division  Technical  Director  for  the  Warfighter  Training  Research 
Division  of  the  Air  Force  Research  Laboratory.  He  received  his  Ph.D.  is  in  Instructional  Systems 
from  Florida  State  University.  His  B.S.  is  in  Psychology  from  Brigham  Young  University. 
Previously  he  worked  as  a  senior  research  psychologist  for  the  Army  Research  Institute  for  the 
Behavioral  and  Social  Sciences  in  Orlando,  Florida.  Prior  to  his  work  with  the  Army  he  was  a 
Research  Psychologist  and  training  analyst  with  the  Naval  Air  Warfare  Center  -  Training 
Systems  Division  in  Orlando,  Florida.  He  is  a  Fellow  in  the  Human  Factors  and  Ergonomics 
Society,  the  American  Psychological  Association,  the  Royal  Aeronautical  Society  of  the  United 
Kingdom,  and  the  Air  Force  Research  Laboratory.  His  research  interests  include:  learning 
organizations,  simulator  design,  flight  training,  advanced  distributed  learning,  accelerated 
learning  and  distributed  mission  training. 


Neil  Chamess 
Florida  State  University 


McGill  University,  Montreal,  Quebec,  Canada 
Carnegie  Mellon  University,  Pittsburgh,  USA 
Carnegie  Mellon  University,  Pittsburgh,  USA 


BA  1965-1969 
MS  1969-1971 
PhD  1969-1974 


Professional  Positions 

1974-1977:  Assistant  Professor,  Wilfrid  Laurier  University,  Canada. 

1977-1993:  Assistant,  Associate,  Full  Professor,  Psychology  Department,  University  of 
Waterloo,  Canada. 

1984-85:  Sabbatical  Fellow,  Mental  Performance  &  Aging  Laboratory,  VA  Outpatient  Clinic, 
Boston 

1990-91:  Visiting  Scholar  at  the  Psychology  Department,  University  of  Victoria,  Victoria,  BC 
1993  (4  mo.):  Visiting  Scientist,  Max  Planck  Institute  for  Human  Development  and  Education, 
Berlin 

1994-I-:  Professor,  Psychology,  Florida  State  University  &  Associate,  Pepper  Institute  on  Aging 
and  Public  Policy.  William  G.  Chase  Professor  of  Psychology 

Editorial  Appointments 

Editor,  Psychology  section,  1985-1989.  Canadian  Journal  on  Aging/Revue  Canadienne  du 
Vieillissement. 

Editorial  board,  1987-1991,  2000-1-  Journal  of  Gerontology,  Psychological  Sciences. 

Editorial  board,  1987-1-  John  Wiley  &  Sons:  Human  Performance  and  Cognition  series. 
Editorial  board,  1992-2001,  2003-1-  Psychology  and  Aging. 

Editorial  Board:  1993-1996  Aging,  Neuropsychology,  and  Cognition. 

Editorial  Board:  1994-1996  Psychological  Bulletin 
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Editorial  Board:  2001+  Gerontechnology 

Editorial  Board:  2007+  Journal  of  Experimental  Psychology:  Applied. 

Honors 

1993  Elected  Eellow,  Canadian  Psychological  Association 

1994  Elected  Eellow,  Gerontological  Society  of  America 

1995  Elected  Eellow,  American  Psychological  Association,  Division  20  (Adult  Development  & 
Aging) 

1997  Elected  Eellow,  American  Psychological  Society 

2008  Appointed  Honorary  Member,  International  Society  of  Gerontechnology. 

Selected  Peer-Reviewed  Publications  (in  chronological  order)  (last  3  years) 

Roring,  R.  W.,  Hines,  E.  G.,  &  Charness,  N.  (2007).  Age  differences  in  identifying  words  in 
synthetic  speech.  Human  Factors,  49,  25-31. 

Roring,  R.W.  &  Charness,  N.  (2007).  A  multilevel  model  analysis  of  expertise  in  chess  across 
the  lifespan.  Psychology  and  Aging,  22,  291-299. 

Jastrzembski,  T.  S.  &  Charness,  N.  (2007).  The  Model  Human  Processor  and  the  older  adult: 
Parameter  estimation  and  validation  within  a  mobile  phone  task.  Journal  of  Experimental 
Psychology:  Applied,  13,  224-248 

Jastrzembski,  T.,  &  Charness,  N.  (2007).  What  older  adults  can  teach  us  about  designing  better 
ballots.  Ergonomics  in  Design,  15,  6-11. 
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Weaver,  S.  J.,  Charness,  N.,  Dijkstra,  K.,  &  Jastrzembski,  T.  (2008).  Musculoskeletal  pain 
prevalence  in  randomly  sampled  university  employees:  Age  and  gender  effectsGerontechnology, 
7,  279-292. 

Charness,  N.  &  Boot,  W.  R.  (in  press).  Aging  and  Information  Technology  Use:  Potential  and 
Barriers.  Current  Directions  in  Psychological  Science. 

Recent  Books  and  Book  Chapters 

Charness,  N.,  Czaja,  S.  J.,  &  Shark,  J.  (2007).  Age  and  technology  for  work.  In  K.  S.  Schulz  & 
G.  A.  Adams  (Eds.)  Aging  and  work  in  the  21st  century  (pp.  225-249).  Mahwah,  NJ.:  Erlbaum. 
Charness,  N.  &  Krampe,  R.  T.  (2008).  Expertise  and  knowledge.  In  D.  E.  Alwin  &  S.  M.  Hofer 
(Eds.)  Handbook  on  Cognitive  Aging:  Interdisciplinary  Perspectives  (pp.  244-258).  Thousand 
Oaks,  California:  Sage. 

Charness,  N.  (2008).  Technology  as  multiplier  effect  for  an  aging  work  force.  In  K.  W.  Schaie 
&  R.  Abeles  (Eds.)  Social  structures  and  aging  individuals:  Continuing  challenges  (pp.  167- 
192).  New  York:  Springer. 

Charness,  N.,  &  Jastrzembski,  T.  S.  (2009).  Gerontechnology.  In  P.  Saariluoma  &  H.  Isomaki 
(Eds.).  Future  Interaction  Design  II  (pp.  1-29),  Eondon:  Springer-Verlag. 
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adults:  Principles  and  creative  human  factors  approaches  (2nd  Ed.).  Boca  Raton:  CRC  Press. 
Charness,  N.  (2009).  Skill  acquisition  in  older  adults:  Psychological  mechanisms.  In  S.  J.  Czaja 
&  J.  Sharit  (Eds.).  Aging  and  work:  Issues  and  implications  in  a  changing  landscape  (pp.  232- 
258).  Baltimore:  The  Johns  Hopkins  University  Press. 
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Paul  J.  Feltovich 

Research  Scientist  at  the  Florida  Institute  for  Human  and  Machine  Cognition 


He  has  conducted  research  and  published  on  topics  such  as  expert-novice  differences  in  complex 
cognitive  skills,  conceptual  understanding  and  misunderstanding  for  complex  knowledge,  and 
novel  means  of  instruction  in  complex  and  ill- structured  knowledge  domains  (Cognitive 
Flexibility  Theory  and  Reductive  Bias  Theory— with  Rand  Spiro  and  Richard  Coulson).  Since 
joining  IHMC  in  2001,  he  has  been  investigating  (with  Jeff  Bradshaw  and  the  KAoS  team) 
coordination,  regulation,  and  teamwork  in  mixed  groups  of  humans  and  intelligent  software 
agents.  These  studies  employ  a  “cultural”  approach  to  controlling  and  modeling  software  agent 
activity,  based  on  the  utilization  of  diverse  policy  systems  ranging  from  formal  pertinent  law,  to 
group  traditions,  to  standards  of  practice,  to  norms  for  acceptable  every  day  behavior  (e.g., 
various  codes  of  etiquette).  The  work  also  addresses  Human- Agent  coordination  in  mixed 
teams  and  factors  that  contribute  to  making  software  agents  acceptable  to  humans  as  partners  in 
complex  and  consequential  work.  He  has  authored  more  than  one  hundred  professional  articles 
and  three  books.  In  particular,  he  is  co-author  (with  Micki  Chi  and  Robert  Glaser)  of  a 
designated  Science  Citation  Classic  paper  on  problem  solving  in  physics  which  has  contributed 
greatly  to  the  development  of  human  expertise  as  a  field  of  study  in  cognitive  science.  Feltovich 
was  chosen  to  write  one  of  the  articles  on  expertise  for  the  Third  International  Encyclopedia  for 
the  Social  and  Behavioral  Sciences,  and  has  recently  co-edited  (with  Anders  Ericsson,  Neil 
Charness,  &  Robert  Hoffman)  the  first  ever  Cambridge  Handbook  on  Expertise  and  Expert 
Performance.  He  is  also  co-editor  (with  Ken  Ford  and  Robert  Hoffman)  of  Expertise  in  Context: 
Human  and  Machine  (AAAI/MIT)  and  (with  Ken  Forbus)  Smart  Machines  in  Education 
(AAAI/MIT). 


Stephen  M.  Fiore 
University  of  Central  Florida 


Education 

Institution _ 

Montgomery  College 
University  of  Maryland 
University  of  Maryland 
University  of  Pittsburgh 
University  of  Pittsburgh 


Major _ 

Business  Administration 
Marketing 
Psychology 
Cognitive  Psychology 
Cognitive  Psychology 


Degree  and  Year 

A. A.  1986 

B. S. 1988 
B.A.  1991 
M.S.  1994 
Ph.D.  2000 


Appointments 

2007 -present  Director,  Cognitive  Sciences  Laboratory,  Institute  for  Simulation  and  Training 

University  of  Central  Florida,  Orlando,  FL 

2005-present  Assistant  Professor,  Cognitive  Sciences  Program,  Department  of  Philosophy, 
University  of  Central  Florida,  University  of  Central  Florida,  Orlando,  FL 
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2002-2007 


2000-present 

1998-2000 

1992-1998 

1990-1992 


Director,  Consortium  for  Research  in  Adaptive  Distributed  Learning 
Environments,  Institute  for  Simulation  and  Training  University  of  Central 
Florida,  Orlando,  FL 

Research  Scientist,  Team  Performance  Faboratory,  Department  of  Psychology, 
University  of  Central  Florida,  Orlando,  FF 

Research  Associate,  Department  of  Psychology  -  Team  Performance 
Faboratory,  University  of  Central  Florida,  Orlando,  FF 
Fab  Coordinator,  Fearning  Research  and  Development  Center,  University  of 
Pittsburgh,  Pittsburgh,  PA 

Research  Assistant,  Faboratory  of  Socio-environmental  Studies,  National 
Institute  of  Mental  Health,  Bethesda,  MD 


Selected  Publications 

Hoffman,  R.  R.,  Feltovich,  P.  J.,  Fiore,  S.  M.,  Klein,  G.,  &  Ziebell,  D.  (2009).  Accelerated 
Fearning  (?).  IEEE  Intelligent  Systems,  24(2),  18-22. 

Fiore,  S.  M.  (2008).  Interdisciplinarity  as  Teamwork:  How  the  Science  of  Teams  can  inform 
Team  Science.  Small  Group  Research,  39(3),  251-211. 

Fiore,  S.M.  &  Salas,  E.  (2008).  Cognition,  Competition,  and  Coordination:  Understanding 
Expertise  in  Sports  and  its  Relevance  to  Fearning  and  Performance  in  the  Military.  Special  Issue 
of  Military  Psychology,  Volume  20,  Issue  SI 

Hoffman,  R.  R.,  &  Fiore,  S.  M.  (2007).  Perceptual  (re)learning:  A  leverage  point  for  human- 
centered  computing.  IEEE  Intelligent  Systems,  22,  3,  79-83. 

Fiore,  S.M.  &  Salas,  E.  (2006).  Team  Cognition  and  Expert  Teams:  Emerging  Insights  into 
Fearning  and  Performance  for  Exceptional  Teams.  International  Journal  of  Sports  and  Exercise 
Psychology,  Volume  4. 

Fiore,  S.  M.  &  Salas,  E.  (Editors).  (2007).  Toward  a  science  of  distributed  learning. 
Washington,  DC:  American  Psychological  Association. 

Fiore,  S.  M.,  Harrison,  G.  W.,  Hughes,  C.  E.,  &  Rutstrom,  E.  E.  (2009).  Virtual  Experiments  and 
Environmental  Policy.  Journal  of  Environmental  Economics  &  Management,  25,  65-86. 

Fidopiastis,  C.M.,  Stapleton,  C.B.,  Whiteside,  J.D.,  Hughes,  C.E.,  Fiore,  S.M.,  Martin,  G.A., 
Rolland,  J.P.  &  Smith  E.M.  (2006).  Human  experience  modeler:  Context  driven  cognitive 
retraining  to  facilitate  transfer  of  training.  Cyberpsychology  &  Behavior,  9,  2,  183-187. 

Cuevas,  H.  M.,  Fiore,  S.  M.,  Bowers,  C.  A.,  &  Salas,  E.  (2004).  Fostering  constructive  cognitive 
and  metacognitive  activity  in  computer-based  complex  task  training  environments.  Computers 
in  Human  Behavior,  20,  225-241. 

Fiore,  S.M.,  Jentsch,  F.,  &  Rubinstein,  J.  (2004).  Science  and  Security:  Human-centered 
Research  for  Keeping  our  Nation  Safe.  Special  Section  of  International  Journal  of  Cognitive 
Technology,  Volume  9. 
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Dr.  J.  D.  Fletcher 
Institute  for  Defense  Analysis 


Dr.  J.  D.  Fletcher  is  a  research  staff  member  at  the  Institute  for  Defense  Analyses  where  he 
specializes  in  manpower,  personnel,  and  training  issues.  His  graduate  degrees  are  in  computer 
science  and  educational  psychology,  both  from  Stanford  University.  He  has  held  academic 
positions  in  psychology,  computer  science,  and  systems  engineering  and  government  positions  as 
a  research  psychologist  and/or  program  manager  for  the  Navy,  Army,  the  Defense  Advanced 
Research  Projects  Agency,  and  the  White  House  Office  of  Science  and  Technology  Policy  where 
he  developed  cross-agency  plans,  programs,  and  policies  in  education  and  training.  He  is  a 
fellow  of  the  American  Educational  Research  Association  and  three  divisions  of  the  American 
Psychological  Association.  His  research  activities  have  produced  intelligent^  tutoring  systems, 
networked  simulations,  wearable  voice-interactive  performance  aids,  international  specifications 
for  sharable  digital  objects,  and  cost-effectiveness  analyses  on  the  use  and  impact  of  instructional 
technology. 


Jared  Freeman.  Ph.D. 

Aptima,  Inc. 

Jared  Freeman,  Ph.D.,  is  Senior  Vice  President  of  Research  at  Aptima.  As  a  member  of  Aptima's 
executive  team,  he  works  to  maintain  the  quality  of  Aptima's  research  and  to  coordinate  research 
efforts  within  the  company. 


Dr.  Freeman  is  a  cognitive  scientist  by  training  with  a  Doctorate  in  Human  Fearning  and 
Cognition.  He  investigates  problem  solving  and  decision  making  in  real-world  settings,  and 
defines  training  and  job  aids  that  address  these  challenges.  Within  the  past  several  years.  Dr. 
Freeman  has  served  as  Aptima’s  Principal  Investigator  in  research  and  development  projects  to 
analyze,  model,  train,  and  aid  information  system  managers,  imagery  analysts,  information 
operations  specialists,  technicians  for  intelligence  equipment,  warfighters  engaged  in  anti-piracy 
operations  at  sea  and  peacekeeping  ashore,  and  interagency  personnel  engaged  in  Maritime 
Domain  Awareness  missions.  He  also  has  served  as  P.I.  on  projects  to  integrate  neuro-cognitive 
and  behavioral  measurement  technologies;  measure,  monitor  and  manage  team  knowledge  and 
collaboration;  automate  the  analysis  of  voice  communications;  automate  the  analysis  of  written 
usability  documents;  and  model  the  fit  between  human  cognitive  abilities,  decision  support 
systems,  and  mission  requirements.  His  business  development  efforts  have  generated  numerous 
contracts  from  research  laboratories,  training,  and  acquisition  organizations  for  the  Army,  Navy, 
Marines,  Air  Force,  and  Joint  organizations. 

Prior  to  joining  Aptima,  Dr.  Freeman  was  a  scientist  at  Cognitive  Technologies,  where  he 
developed  a  highly  automated  approach  to  assessing  student  knowledge  of  tactical  situations,  as 
well  as  instructional  technology  that  improved  situation  assessment  and  planning  by  Army 
officers.  Dr.  Freeman  consulted  to  Bell  Fabs  for  two  years,  where  he  conducted  research 
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concerning  how  experts  diagnose  failures  in  complex  software  systems,  co-developed  a  course 
based  on  that  research,  and  taught  it  nationally. 

Dr.  Freeman  has  published  more  than  100  chapters,  articles,  and  proceedings  papers  on  the 
topics  of  critical  thinking,  task  analysis,  computational  modeling  of  teams,  performance 
measurement,  and  training. 

Dr.  Freeman  received  a  Ph.D.  from  Columbia  University  and  a  M.A.  in  Educational  Technology 
from  Teachers  College,  Columbia  University.  He  is  a  member  of  the  Human  Factors  and 
Ergonomics  Society,  and  a  contributing  editor  to  Human  Factors. 


Dr.  Andrew  S.  Gibbons 
Brigham  Young  University 


Andy  Gibbons  is  the  chair  of  Instructional  Psychology  and  Technology  at  Brigham  Young 
University.  Prior  to  that,  he  was  a  faculty  member  at  Utah  State  University  for  10  years.  This 
followed  18  years  of  leading  instructional  design  projects  in  industry,  including  work  on  large- 
scale  training  development,  design  of  simulations,  and  innovative  forms  of  computer-based 
instruction.  Dr.  Gibbons’  research  focuses  on  the  architecture  of  instructional  designs.  He  has 
published  a  design  theory  of  Model-Centered  Instruction,  proposed  a  general  Eayering  Theory  of 
instructional  designs,  and  is  currently  studying  the  use  of  design  languages  in  relation  to  design 
layers  to  create  instructional  systems  that  are  adaptive,  generative,  and  scalable. 

Waters,  S.  H.  &  Gibbons,  A.  S.  (2004).  Design  languages,  notation  systems,  and  instructional 
technology:  A  case  study.  Educational  technology  research  and  development,  52(2),  57-68. 

Gibbons,  A.  S.  (2003).  What  and  how  do  designers  design:  A  theory  of  design  structure.  Tech 
trends,  47(5),  22-21 . 

Eairweather,  P.  G.  &  Gibbons,  A.  S.  (2000).  Distributed  learning:  Two  steps  forward,  one  back? 
One  forward,  two  back?  IEEE  Concurrency,  8(2),  8-9-I-. 

Gibbons,  A.  S.,  Bhardwaj,  K.  K.,  &  Richards,  R.  (1998).  The  single-parse  method  of  design  for 
problem-based  instruction.  Educational  technology  research  and  development,  46(2),  110-116. 

Gibbons,  A.  S.,  Bunderson,  C.  V.,  Olsen,  J.  B.,  &  Robertson,  J.  (1995).  Work  models:  still 
beyond  instructional  objectives.  Machine-mediated  learning,  5(3&4),  221-236. 

Gibbons,  A.  S.  &  Rogers,  P.  C.  (2009).  The  architecture  of  instructional  theory.  In  C.  M. 
Reigeluth  &  A.  Carr-Chellman  (Eds.),  Instructional-design  theories  and  models,  Volume  III. 
Mahwah,  NJ:  Eawrence  Erlbaum  Associates. 

Gibbons,  A.  S.,  McConkie,  M.,  Seo,  K.  K.,  &  Wiley,  D.  (2009).  Theory  for  the  design  of 
instructional  simulations  and  microworlds.  In  C.  M.  Reigeluth  and  A.  Carr-Chellman  (Eds.), 
Instructional-design  theories  and  models,  Volume  III.  Mahwah,  NJ:  Eawrence  Erlbaum 
Associates. 
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Gibbons,  A.  S.  (2008).  Coming  at  design  from  a  different  angle:  Functional  design.  In  L.  Moller, 
J.  Huett  &  D.  Harvey  (Eds.),  Learning  and  instructional  technologies  for  the  21st  century.  New 
York:  Springer. 

Gibbons,  A.  S.  (2006).  A  model  of  technology  capable  of  generating  research  questions.  2006 
Educational  technology  and  media  yearbook.  (In  conjunction  with  E.  Boling  and  B. 
Bichelmeyer).  Bloomington,  IN:  Association  for  Educational  Communications  and  Technology. 
Also  presented  at  the  2004  Association  for  Educational  Communications  and  Technology 
conference. 


Stephen  E.  Goldberg.  Ph.D. 

U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences  (ARI) 

Dr.  Stephen  E.  Goldberg  has  over  thirty-four  years  experience  as  a  U.S.  Army  Research  Institute 
for  the  Behavioral  and  Social  Sciences  (ARI)  Research  Psychologist.  Dr.  Goldberg  received  a 
doctorate  in  Cognitive  Psychology  from  the  State  University  of  New  York  at  Buffalo  in  1974. 
He  has  served  as  an  ARI  researcher  at  locations  in  Alexandria,  VA,  and  Et.  Knox,  KY.  In  1984 
he  became  ARTs  liaison  to  the  U.S.  Army’s  Training  and  Doctrine  Command  (TRADOC),  Et. 
Monroe,  VA.  Dr.  Goldberg  assumed  his  current  assignment  as  the  Chief  of  ARI's  Orlando 
(Advanced  Training  Technology)  Research  Unit  in  1989.  He  supervises  a  research  program 
focused  on  the  behavioral  aspects  of  training,  with  emphasis  on  feedback  processes  and  training 
and  performance  in  virtual  simulations  and  games. 

Dr.  Goldberg  served  for  five  years  as  Chairman  and  U.S.  National  Deader  for  the  Training 
Technology  Panel  of  The  Technical  Cooperation  Program  (TTCP),  an  international  program  for 
data  exchange  and  collaborative  research  between  the  United  Kingdom,  Canada,  Australia,  New 
Zealand,  Australia,  and  the  United  States.  Dr.  Goldberg  has  also  held  leadership  role  in  three 
NATO  research  study  groups  investigating  the  applicability  of  virtual  environments  to  military 
operations  and  training.  Dr.  Goldberg  is  a  past  President  and  of  the  Society  for  Military 
Psychology  (Division  19)  of  the  American  Psychological  Association.  He  is  a  Eellow  of  the 
American  Psychological  Association  and  the  American  Educational  Research  Association. 

Dr.  Goldberg  has  authored  numerous  technical  reports,  book  chapters,  and  conference 
presentations.  His  research  interests  include  training  methods,  skill  retention,  use  of  virtual 
reality  for  training,  and  performance  measurement. 
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Frank  L.  Greitzer.  Ph.D. 

Pacific  Northwest  National  Laboratory 


Frank  L.  Greitzer,  Ph.D.,  is  a  Chief  Scientist  at  the  Pacific  Northwest  National  Laboratory 
(PNNL),  where  he  conducts  R&D  in  human-information  interaction  for  diverse  problem 
domains.  He  holds  a  PhD  degree  in  Mathematical  Psychology  with  specialization  in  memory 
and  cognition  and  a  BS  degree  in  Mathematics.  Dr.  Greitzer  leads  a  R&D  focus  area  of 
Cognitive  Informatics  that  addresses  human  factors  and  social/behavioral  science  challenges 
through  modeling  and  advanced  engineering/computing  approaches.  With  over  thirty  years  of 
applied  research  and  development  experience  in  cognitive  psychology,  human  information 
processing,  and  user-centered  design,  his  research  interests  include  modeling  human  behavior 
with  application  to  identifying/predicting  malicious  insider  cyber  activities,  modeling  socio¬ 
cultural  factors  as  predictors  of  terrorist  activities,  and  human  information  interaction  concepts 
for  enhancing  decision  making  in  domains  such  as  intelligence  analysis  or  electric  power  grid 
operations.  His  research  interests  also  include  evaluation  methods  and  metrics  for  assessing 
effectiveness  of  decision  aids,  analysis  methods  and  displays.  In  the  area  of  cyber  security.  Dr. 
Greitzer  serves  as  Predictive  Defense  Focus  Area  Lead  for  the  PNNL  Information  and 
Infrastructure  Integrity  Initiative.  Dr.  Greitzer  also  has  conducted  research  to  improve  training 
effectiveness  by  applying  cognitive  principles  in  innovative,  interactive,  scenario-based  training 
and  serious  gaming  approaches.  Many  of  his  publications  and  descriptions  of  representative 
projects  may  be  found  at  the  Cognitive  Informatics  web  site,  http://www.pnl.gov/cogInformatics. 
In  addition  to  his  work  at  PNNL,  Dr.  Greitzer  serves  as  an  adjunct  faculty  member  at 
Washington  State  University,  Tri-Cities  campus,  where  he  teaches  courses  for  the  computer 
science  department  (interaction  design)  and  for  the  psychology  department  (human  factors).  Dr. 
Greitzer  also  serves  on  the  Editorial  Board  of  the  Journal  of  Cognitive  Informatics  &  Natural 
Intelligence. 
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Alice  F.  Healy 
University  of  Colorado 


Education 

Vassar  College,  Poughkeepsie,  New  York,  A.B.,  1968,  Psychology 

The  Rockefeller  University,  New  York,  New  York,  Ph.D.,  1973,  Psychology 

Positions  and  Employment 

1968-1973  USPHS  Training  Grant  Fellow,  The  Rockefeller  University, 

1968-1973  1973-1978  Assistant  Professor  of  Psychology,  Yale  University 

1978- 1981  Associate  Professor  of  Psychology,  Yale  University 

1981-1984  Associate  Professor  of  Psychology,  University  of  Colorado,  Boulder  1984-present 
Professor  of  Psychology  and  Neuroscience,  University  of  Colorado,  Boulder  2007-present 
College  Professor  of  Distinction,  University  of  Colorado,  Boulder 

Other  Experience  and  Professional  Memberships 

1979- 1981  Member,  NIMH  Basic  Behavioral  Processes  Research  Review  Committee  1976-1980 
Research  Associate,  Haskins  Laboratories  1981-1984 

Associate  Editor,  Journal  of  Experimental  Psychology:  Eearning,  Memory,  and  Cognition  1985- 
1989 

Editor,  Memory  &  Cognition  1987-1992 

Member  of  the  Governing  Board,  Psychonomic  Society  1994-1995 
President  of  Rocky  Mountain  Psychological  Association  1995-1996 

Chair  of  Electorate  J  (Psychology),  American  Association  for  the  Advancement  of  Science  2004- 
2005 

President  of  Division  3  (Experimental  Psychology),  American  Psychological  Association 
2006-present  Director,  Center  for  Research  on  Training,  University  of  Colorado,  Boulder 
2008-2009  Chair,  Society  of  Experimental  Psychologists 

Honors 

1987-1988  James  McKeen  Cattell  Eund  Sabbatical  Award  2005  Women  in  Cognitive  Science 
Mentorship  Award  2006  Rocky  Mountain  Psychological  Association  Distinguished  Service 
Award 

Selected  Publications 

Schneider,  V.  I.,  Healy,  A.  E.,  &  Bourne,  E.  E.,  Jr.  (2002).  What  is  learned  under  difficult 
conditions  is  hard  to  forget:  Contextual  interference  effects  in  foreign  vocabulary  acquisition, 
retention,  and  transfer.  Journal  of  Memory  and  Eanguage,  46,  419-440. 

Healy,  A.  E.,  Kole,  J.  A.,  Buck-Gengler,  C.  J.,  &  Bourne,  E.  E.,  Jr.  (2004).  Effects  of  prolonged 
work  on  data  entry  speed  and  accuracy.  Journal  of  Experimental  Psychology:  Applied,  10,  188- 
199. 
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Healy,  A.  F.,  Wohldmann,  E.  L.,  Parker,  J.  T.,  &  Bourne,  L.  E.,  Jr.  (2005).  Skill  training, 
retention,  and  transfer:  The  effects  of  a  concurrent  secondary  task.  Memory  &  Cognition,  33, 
1457-147E 


Healy,  A.  E.,  Wohldmann,  E.  E.,  Sutton,  E.  M.,  &  Bourne,  E.  E.,  Jr.  (2006).  Specificity  effects  in 
training  and  transfer  of  speeded  responses.  Journal  of  Experimental  Psychology:  Eearning, 
Memory,  and  Cognition,  32,  534-546. 

Kole,  J.  A.,  &  Healy,  A.  E.  (2007).  Using  prior  knowledge  to  minimize  interference  when 
learning  large  amounts  of  information.  Memory  &  Cognition,  35,  124-137. 

Wohldmann,  E.  E.,  Healy,  A.  E.,  &  Bourne,  E.  E.,  Jr.  (2007).  Pushing  the  limits  of  imagination: 
Mental  practice  for  learning  sequences.  Journal  of  Experimental  Psychology:  Eearning,  Memory, 
and  Cognition,  33,  254-261. 

Healy,  A.  E.,  Shea,  K.  M.,  Kole,  J.  A.,  &  Cunningham,  T.  E.  (2008).  Position  distinctiveness, 
item  familiarity,  and  presentation  frequency  affect  reconstruction  of  order  in  immediate  episodic 
memory.  Journal  of  Memory  and  Eanguage,  58,  746-764. 

Kole,  J.  A.,  Healy,  A.  E.,  &  Bourne,  E.  E.,  Jr.  (2008).  Cognitive  complications  moderate  the 
speed-accuracy  tradeoff  in  data  entry:  A  cognitive  antidote  to  inhibition.  Applied  Cognitive 
Psychology,  22,  917-937. 

Wohldmann,  E.  E.,  Healy,  A.  E.,  &  Bourne,  E.  E.,  Jr.  (2008).  A  mental  practice  superiority 
effect:  Eess  retroactive  interference  and  more  transfer  than  physical  practice.  Journal  of 
Experimental  Psychology:  Eearning,  Memory,  and  Cognition,  34,  823-833. 

Wohldmann,  E.  E.,  Healy,  A.  E.,  &  Bourne,  E.  E.,  Jr.  (2008).  Global  inhibition  and  midcourse  corrections 
in  speeded  aiming.  Memory  &  Cognition,  36,  1228-1235. 
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Lieutenant  Colonel  Gina  Lee  Hilger 
U.S.  Air  Force 


Lieutenant  Colonel  Gina  L.  Hilger  is  the  Chief  of  the  Intelligence,  Surveillance,  and 
Reconnaissance  (ISR)  Requirements  and  Operations  Division,  Intelligence  Directorate,  US  Air 
Forces  Central,  Shaw  Air  Force  Base,  South  Carolina.  She  is  responsible  for  providing 
collection  management,  multi-intelligence  exploitation,  and  processing,  exploitation,  and 
dissemination  support  to  combat  operations  in  the  US  Central  Command  AOR. 

Lieutenant  Colonel  Hilger  was  born  in  Belleville,  IL  on  12  September  1971  and  received  her 
commission  in  1993  through  the  United  States  Air  Force  Academy. 

Education 

1993  Bachelor  of  Science  degree  in  Political  Science/Legal  Studies  and  Spanish  minor,  US  Air 
Force  Academy 

1995  Master’s  degree  in  International  Security  and  Economic  Policy,  University  of  Maryland 

1996  Intelligence  Officer  Training  Course,  Goodfellow  AFB,  TX 
1998  Squadron  Officer  School,  Maxwell  AFB,  AL 

2001  Master’s  degree  in  Strategic  Management,  George  Washington  University 
2003  Air  Command  and  Staff  College  (Distance  Learning) 

2005  Master’s  degree  in  Counseling  and  Leadership,  University  of  Colorado-Colorado  Springs 
2008  Air  War  College  (Distance  Learning) 

Assignments 

1 .  Aug  93-May  95,  School  of  Public  Affairs  fellowship.  University  of  Maryland,  College  Park, 
MD 

2.  Jun  95-Feb  96,  Intelligence  Officer  Training  Course,  Goodfellow  AFB,  TX 

3.  Feb  96-Oct  96,  Deputy  Chief,  5th  Air  Force  Bilateral  Plans  &  Programs,  Yokota  AB,  Japan 

4.  Oct  96-Jan  97,  Liaison  Officer  &  Command  Briefer,  ACOC,  JTF-SWA,  Riyadh,  Saudi 
Arabia 

5.  Feb  97-Dec  97,  Chief,  Combat  Ops  Intel  Division,  605  AIF,  5th  Air  Force,  Yokota  AB,  Japan 

6.  Jan  98-Feb  98,  Squadron  Officer  School,  Maxwell  AFB,  AL 

7.  Mar  98-Jul  99,  Chief,  Operations  Intel  Element,  8th  Operations  Support  Squadron,  Kunsan 
AB,  South  Korea 

8.  Jul  99- Jun  01,  Air  Eorce  Intern,  Air  Eorce  Intern  Program  (AEIP),  Pentagon,  Washington,  DC 

9.  Jun  01 -Mar  03,  Intel  Plight  Commander,  39th  Operations  Support  Squadron,  Incirlik  AB, 
Turkey 

10.  Apr  03-May  04,  Course  Director,  Intel  Master  Skills  Course,  315th  Training  Squadron, 
Goodfellow  APB,  TX 

11.  May  04-May  05,  APIT/IDE  Student,  University  of  Colorado-Colorado  Springs,  CO 

12.  May  05-Jul  07,  Squadron  Commander/Air  Officer  Commanding  (AOC),  Squadron  Six,  CG- 
01,  USAPA,  CO 

13.  Jul  07-Nov  07,  Chief,  Current  Intel  Branch,  US  Air  Porce  Space  Command,  Peterson  APB, 
CO 

14.  Nov  07-Nov  08,  Director  of  Operations,  732d  Expeditionary  Intel  Squadron,  Balad  AB,  Iraq 
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15.  Feb  09-present,  Chief,  ISR  Requirements  and  Operations  Division,  9  AF/AFCENT,  Shaw 
AFB,  SC 


Major  Awards  And  Decorations 

Outstanding  Graduate,  Air  War  College,  Distance  Learning,  2008 

Meritorious  Service  Medal,  oak  leaf  cluster 

39th  Operations  Group  CGO  of  the  Year,  2002 

7  AF  /  8  FW  Lance  P.  Sijan  Award,  1999 

5  AL  Intel  Officer  of  the  Year,  1997 

Joint  Service  Achievement  Medal,  oak  leaf  cluster 

Air  Lorce  Commendation  Medal,  2  x  oak  leaf  clusters 

Honor  Graduate,  Intel  Officer  Training  Course,  1996 

Co- Valedictorian,  Master’s  in  Public  Management  Program,  Univ  of  MD,  1995 
#1  Academic  Graduate  (1/963)  and  Distinguished  Graduate  (4/963),  United  States  Air  Lorce 
Academy,  1993 


Effective  Dates  Of  Promotion 


Second  Lieutenant 
Lirst  Lieutenant 
Captain 
Major 

Lieutenant  Colonel 


2  Jun  1993 
2  Jun  1995 
2  Jun  1997 
1  Aug  2003 
1  Aug  2008 
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Robert  R.  Hoffman.  Ph.D. 

Institute  for  Human  and  Machine  Cognition 


Hoffman  is  recognized  as  one  of  the  world  leaders  in  the  field  of  cognitive  systems  engineering 
and  Human-Centered  Computing.  He  is  a  Fellow  of  the  Association  for  Psychological  Science 
and  a  Fulbright  Scholar.  His  Ph.D.  is  in  experimental  psychology  from  the  University  of 
Cincinnati,  where  he  received  McMicken  Scholar,  Psi  Chi,  and  Delta  Tau  Kappa  Honors. 
Following  a  Postdoctoral  Associateship  at  the  Center  for  Research  on  Human  Learning  at  the 
University  of  Minnesota,  Hoffman  joined  the  faculty  of  the  Institute  for  Advanced  Psychological 
Studies  at  Adelphi  University.  He  began  his  career  as  a  psycholinguist,  and  founded  the  journal. 
Metaphor  and  Symbol.  His  subsequent  research  leveraged  the  psycholinguistics  background  in 
the  study  of  methods  for  eliciting  the  knowledge  of  domain  experts.  Hoffman  has  been 
recognized  internationally  in  disciplines  including  psychology,  remote  sensing,  weather 
forecasting,  and  artificial  intelligence,  for  his  research  on  human  factors  in  remote  sensing,  his 
work  in  the  psychology  of  expertise  and  the  methodology  of  cognitive  task  analysis,  and  for  his 
work  on  HCC  issues  intelligent  systems  technology  and  the  design  of  macrocognitive  work 
systems.  Hoffman  is  a  Co-Editor  for  the  Department  on  Human-Centered  Computing  in  IEEE: 
Intelligent  Systems.  He  is  Editor  for  the  book  Series,  "Expertise:  Research  and  Applications."  He 
is  a  co-founder  and  Track  Editor  for  the  Journal  of  Cognitive  Engineering  and  Decision  Making. 
His  major  current  projects  involve  evaluating  the  effectiveness  of  knowledge  management,  and 
performance  measurement  for  macrocognitive  work  systems.  A  full  vita  and  all  of  his 
publications  are  available  for  download  at  [www.ihmc.us/users/rhoffman/main]. 

Recent  Books 

Hoffman,  R.  R.  and  Militello,  E.  G.  (2008).  Perspectives  on  Cognitive  Task  Analysis:  Historical 
Origins  and  Modern  Communities  of  Practice.  Boca  Raton,  EE:  CRC  Press/Taylor  and  Erancis. 

Crandall,  B.,  Klein,  G.,  &  Hoffman,  R.  (2007).  Working  minds:  A  practitioner's  guide  to  cognitive 
task  analysis.  Cambridge  MA:  MIT  Press. 

Ericsson,  K.  A.,  Charness,  N.,  Eeltovich,  P.  J.  and  Hoffman,  R.  R.,  (Eds.)  (2006).  Cambridge 
handbook  of  expertise  and  expert  performance.  New  York:  Cambridge  University  Press. 


Selected  Publications 

Hoffman,  R.  R.,  Marx,  M.,  Amin,  R.,  &  McDermott,  P.  (2010).  Measurement  for  evaluating  the 
learnability  and  resilience  of  methods  of  cognitive  work.  Theoretical  Issues  in  Ergonomic 
Science,  in  press. 

Hoffman,  R.  R.,  Eee,  J.  D.,  Woods,  D.  D.,  Shadbolt,  N.,  Miller,  J.  &  Bradshaw,  J.  M.  (2009, 
November/December).  The  dynamics  of  trust  in  cyberdomains.  IEEE  Intelligent  Systems,  pp.  5- 
11. 

Hoffman,  R.  R.,  Eiore,  S.  M.,  Klein,  G.,  Eeltovich,  P.  and  Ziebell,  D.  (2009,  March/April). 
Accelerated  learning  (?).  IEEE:  Intelligent  Systems,  pp.  18-22. 
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Hoffman,  R.  R.,  Neville,  K.  N.  and  Fowlkes,  J.  (2009).  Using  cognitive  task  analysis  to  explore 
issues  in  the  procurement  of  intelligent  decision  support  systems.  Cognition,  Technology,  and 
Work,  11,  57-70. 

Hoffman,  R.  R.  (2008).  Human  factors  contributions  to  knowledge  elicitation.  Human  Factors 
(Special  50th  Anniversary  Issue),  50,  481-488. 

Hoffman,  R.  R.,  Ziebell,  D.,  Fiore,  S.  M.  and  Becerra-Femandez,  I.  (2008,  May/June).  Knowledge 
management  revisited.  IEEE:  Intelligent  Systems,  pp.  84-88. 

Klein,  G.,  Moon,  B.,  &  Hoffman,  R.  R.  (2006,  July/August).  Making  sense  of  sensemaking.  IEEE 
Intelligent  Systems,  pp.  22-26. 

Hoffman,  R.  R.,  &  Shattuck,  L.  G.  (2006/March- April).  Should  we  rethink  how  we  do  OPORDs? 
Military  Review,  pp.  100-107. 

Hoffman,  R.  R.,  Coffey,  J.  W.,  Ford,  K.  M.,  &  Novak,  J.  D.  (2006).  A  method  for  eliciting, 
preserving,  and  sharing  the  knowledge  of  forecasters.  Weather  and  Forecasting,  21,  416-428. 

Hoffman,  R.  R.,  &  Woods,  D.  D.  (January/February,  2005).  Steps  toward  a  theory  of  complex  and 
cognitive  systems.  IEEE:  Intelligent  Systems,  pp.  76-79. 

Hoffman,  R.  R.,  &  Deffenbacher,  K.  A.  (1993).  An  analysis  of  the  relations  of  basic  and  applied 
science.  Ecological  Psychology,  5,  315-352. 


Robert  C.  Hubal 
RTI  International  Research.  Inc. 

Robert  Hubal,  senior  research  psychologist  in  RTFs  Substance  Abuse  Epidemiology  and 
Military  Behavioral  Health  Program,  conducts  wide-ranging  research  generally  focusing  on  the 
intelligent  use  of  advanced  technologies  for  training,  work  performance,  and  assessment.  Dr. 
Hubal  has  research  experience  developing  embodied  conversational  agents  for  interaction  skills 
training  and  situated  assessment,  conducting  cost-effectiveness  studies  of  virtual  reality  systems, 
conducting  expertise  and  linguistic  codability  studies,  investigating  adaptive  intelligent  tutoring, 
studying  driver  training  and  assessment,  and  investigating  visual  analytics  and  representation.  He 
has  applied  research  results  to  such  everyday  domains  as  medical  informed  consent,  consumer 
decision  making,  driving,  and  survey  nonresponse  training. 

PhD,  Cognitive  Psychology,  Duke  University,  1996. 

MS,  Computer  Science,  North  Carolina  State  University,  1992. 

SB,  Computer  Science  and  Engineering,  Massachusetts  Institute  of  Technology,  1987. 
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Professional  Experience 

1996  to  date  RTI  International,  Research  Triangle  Park,  NC. 

1992  to  1996,  2000  Duke  University,  Durham,  NC. 

Selected  Book  Chapters 

Eriksson,  E.J.,  Rodman,  R.D.,  &  Hubal,  R.C.  (2007).  Emotions  in  speech:  Juristic  implications. 
In  C.  Muller  (Ed.),  Speaker  classification  I:  Fundamentals,  features,  and  methods  (pp.  152-173). 
Heidelberg,  Germany:  Springer-Verlag. 

Hubal,  R.,  Kizakevich,  P.,  &  Eurberg,  R.  (2007).  Synthetic  characters  in  health-related 
applications.  In  S.  Vaidya,  E.C.  Jain,  &  H.  Yoshida  {FdtS,.),  Advanced  computational  intelligence 
paradigms  in  healthcare  2  (pp.  5-26).  Heidelberg,  Germany:  Springer-Verlag. 

Kizakevich,  P.N.,  Culwell,  A.,  Eurberg,  R.,  Gemeinhardt,  D.,  Grantlin,  S.,  Hubal,  R.,  Stafford, 
A.,  &  Dombroski,  R.T.  (2007).  Virtual  simulation-enhanced  triage  training  for  Iraqi  medical 
personnel.  In  J.D.  Westwood,  R.S.  Haluck,  H.M.  Hoffman,  G.T.  Mogel,  R.  Phillips,  R.A.  Robb, 
&  K.G.  Vosburgh  (Eds.),  In  vivo,  in  vitro,  in  silico:  Designing  the  next  in  medicine  (pp.  223- 
228).  Amsterdam:  lOS  Press. 

Armsby,  P.P.,  Eink,  M.W.,  Hubal,  R.,  Guinn,  C.I.,  Elicker,  E.,  &  Caspar,  R.  (2003).  Accessibility 
and  acceptance  of  a  virtual  respondent-based  interviewer  training  application.  In  R.  Banks  et  al. 
(Eds.),  Survey  and  statistical  computing  IV:  The  impact  of  technology  on  the  survey  process. 
Chesham,  Bucks,  UK:  Association  for  Survey  Computing. 

Hubal,  R.C.,  Kizakevich,  P.,  Guinn,  C.I.,  Merino,  K.D.,  &  West,  S.E.  (2000).  The  virtual 
standardized  patient:  Simulated  patient-practitioner  dialogue  for  patient  interview  training.  In 
J.D.  Westwood,  H.M.  Hoffman,  G.T.  Mogel,  R.A.  Robb,  &  D.  Stredney  (Eds.),  Envisioning 
healing:  Interactive  technology  and  the  patient-practitioner  dialogue  (pp.  133-138).  Amsterdam: 
lOS  Press. 

Selected  Peer-Reviewed  Journal  Articles 

Eishbein,  H.A.,  Sheppard,  M.E.,  Hyde,  C.,  Hubal,  R.C.,  Newlin,  D.B.,  Serin,  R.,  Chrousos,  G.,  & 
Alesci,  S.  (in  press).  Neurocognitive  and  emotional  regulatory  deficits  predict  treatment  response 
in  prison  inmates.  Behavior  and  the  Eaw. 

Hubal,  R.  (2008).  Embodied  tutors  for  interaction  skills  simulation  training.  International  Journal 
of  Virtual  Reality,  7(1),  1-8. 

Hubal,  R.C.,  Evens,  N.R,  EitzGerald,  D.P.,  Hardy,  K.K.,  Willard,  V.W.,  &  Bonner,  M.J.  (2008). 
Implementation  of  EACS  Training  Systems  Association. 

Erank,  G.A.,  &  Hubal,  R.C.  (2008).  Bias  inoculation  advanced  simulation  (BIAS)  training. 
Proceedings  of  the  Behavior  Representation  in  Modeling  and  Simulation  conference.  Orlando, 
PE:  Simulation  Interoperability  Standards  Organization. 

Hubal,  R.,  &  Prank,  G.  (2008).  Enticing  mistakes:  A  strategy  within  simulation  training  of  soft 
skills.  Proceedings  of  the  Interservice/Industry  Training,  Simulation  and  Education  Conference 
(pp.  57-66).  Arlington,  VA:  National  Training  Systems  Association. 

Krizowsky,  P.,  Waters,  H.,  Wright,  M.,  Hubal,  R.,  &  Prank,  G.  (2008).  Dynamically  configured 
scenarios  for  training  adaptive  network  system  operators.  Proceedings  of  the 
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Interservice/Industry  Training,  Simulation  and  Education  Conference  (pp.  1392-1399). 
Arlington,  VA:  National  Training  Systems  Association. 

Frank,  G.,  Hubal,  R.,  &  O’Bea,  M.  (2007).  Using  competency  definitions  to  adapt  training  for 
mission  success.  Proceedings  of  the  Interservice/Industry  Training,  Simulation  and  Education 
Conference  (pp.  1262-1270).  Arlington,  VA:  National  Defense  Industrial  Association.for 
synthetic  characters  for  use  in  studying  facial  expression  recognition  by  survivors  of  childhood 
cancer.  Annual  Review  of  CyberTherapy  and  Telemedicine,  6,  35-40. 

Hubal,  R.C.,  Fishbein,  D.H.,  Sheppard,  M.S.,  Eldreth,  D.E.,  &  Hyde,  C.T.  (2008).  How  do  varied 
populations  interact  with  embodied  conversational  agents?  Findings  from  inner-city  adolescents 
and  prisoners.  Computers  in  Human  Behavior,  24(3),  1104-1138. 

Paschall,  M.J.,  Fishbein,  D.H.,  Hubal,  R.C.,  &  Eldreth,  D.  (2005).  Psychometric  properties  of 
virtual  reality  vignette  performance  measures:  A  novel  approach  for  assessing  adolescents’  social 
competency  skills.  Health  Education  Research:  Theory  and  Practice,  20(1),  61-70. 

Hubal,  R.C.,  Deterding,  R.R.,  Frank,  G.A.,  Schwetzke,  H.F.,  &  Kizakevich,  P.  (2003).  Fessons 
learned  in  modeling  virtual  pediatric  patients.  Studies  in  Health  Technology  and  Informatics,  94, 
I27-I30. 

Selected  Presentations,  Reports,  and  Proceedings 

Hedge,  J.,  &  Hubal,  R.  (2009).  Job  proficiency  work  sample  testing  for  critical  DHS  jobs  and  job 
tasks.  Research  Brief  Pre-Project  Report  prepared  for  the  Institute  for  Homeland  Security 
Solutions,  Research  Triangle  Park,  NC. 

Hubal,  R.C.  (2009).  Mental  models  for  effective  training:  Comparing  expert  and  novice 
maintainers’  mental  models.  Research  Report  1898.  Arlington,  VA:  U.S.  Army  Research 
Institute  for  the  Behavioral  and  Social  Sciences. 

Frank,  G.,  Evens,  N.,  Hubal,  R.,  &  Whiteford,  B.  (2008).  Automated,  interactive  AARs:  A 
positive  spin.  Proceedings  of  the  Interservice/Industry  Training,  Simulation  and  Education 
Conference  (pp.  67-76).  Arlington,  VA:  National 

Frank,  G.,  Whiteford,  B.,  Hubal,  R.,  Sonker,  P.,  Perkins,  K.,  Arnold,  P.,  Presley,  T.,  Jones,  R.,  & 
Meeds,  H.  (2004).  Performance  assessment  for  distributed  learning  using  after  action  review 
reports  generated  by  simulations.  Proceedings  of  the  Interservice/Industry  Training,  Simulation 
and  Education  Conference  (pp.  808-817).  Arlington,  VA:  National  Training  Systems 
Association. 

Guinn,  C.,  &  Hubal,  R.  (2003).  Extracting  emotional  information  from  the  text  of  spoken  dialog. 
Proceedings  of  the  Workshop  on  Assessing  and  Adapting  to  User  Attitudes  and  Affect:  Why, 
When  and  How?  Johnstown,  PA:  User  Modeling,  Inc. 

Maggart,  F.E.,  &  Hubal,  R.C.  (1998).  A  situation  awareness  model.  In  S.E.  Graham  &  M.D. 
Matthews  (Eds.),  Infantry  situation  awareness:  Papers  from  the  1998  Infantry  Situation 
Awareness  Workshop  (pp.  19-28).  Alexandria,  VA:  U.S.  Army  Research  Institute. 
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Gary  Klein 

Macrocognition  LLC  and  Klein  Associates  Division  of  ARA,  Inc. 


Gary  Klein,  Ph.D.  is  a  Senior  Scientist  at  MacroCognition  LLC,  and  also  at  Applied  Research 
Associates.  He  splits  his  time  between  the  two  companies,  working  52%  at  MarcoCognition 
LLC,  and  48%  at  ARA.  He  was  instrumental  in  founding  the  field  of  Naturalistic  Decision 
Making. 

Dr.  Klein  received  his  Ph.D.  in  experimental  psychology  from  the  University  of  Pittsburgh  in 
1969.  He  was  an  Assistant  Professor  of  Psychology  at  Oakland  University  (1970-1974)  and 
worked  as  a  research  psychologist  for  the  U.S.  Air  Force  (1974-1978).  The  R&D  company  he 
founded  in  1978,  Klein  Associates,  was  acquired  by  ARA  in  2005. 

He  has  written:  Sources  of  Power:  How  People  Make  Decisions  (1998);  The  Power  of  Intuition 
(2004)  and  Working  Minds:  A  practitioner’ s  guide  to  Cognitive  Task  Analysis  (Crandall,  Klein, 
&  Hoffman,  2006).  Dr.  Klein’s  next  book.  Streetlights  and  Shadows:  Searching  for  the  keys  to 
adaptive  decision  making,  is  due  out  in  November  2009. 

Dr.  Klein  developed  a  Recognition-Primed  Decision  (RPD)  model  to  describe  how  people 
actually  make  decisions  in  natural  settings.  He  also  developed  methods  of  Cognitive  Task 
Analysis  for  uncovering  the  tacit  knowledge  that  goes  into  decision  making.  He  was  selected  as  a 
Fellow  of  Division  19  of  the  American  Psychological  Association  in  2006,  and  in  2008  he 
received  the  Jack  A.  Kraft  Innovator  Award  from  the  Human  Factors  and  Ergonomics  Society. 


Lochlan  E.  Magee 

Defence  Research  and  Development  Canada 

Dr.  Magee  graduated  from  the  University  of  Toronto  with  a  PhD  in  1982  in  experimental 
psychology,  with  specialization  in  human  information  processing.  He  received  National 
Research  Council  of  Canada  and  Ontario  Graduate  Student  scholarships  during  his  studies. 

In  1980  he  joined  Defence  Research  and  Development  Canada  as  a  Defence  Scientist  (DS)  and 
became  responsible  for  the  conduct  and  management  of  R&D  activities  associated  with  the 
design,  use  and  evaluation  of  simulators  for  the  CP. 

His  major  career  accomplishments  include  the  generation  and  application  of  human  factors 
knowledge  in  the  design  and  assessment  of  low-cost  simulators.  These  include  invention  of 
training  simulators  for  the  land  (Eeopard  Cl  tank  gunnery  trainer  &  TOW  guided  missile 
simulator),  maritime  (submarine  periscope  simulator  &  officer-of-the- watch  simulator)  and  air 
(Sea  King  helicopter)  environments.  Dr  Magee  has  also  patented  enabling  technology  (i.e.,  an 
interactive  video  processor),  investigated  unwanted  side  effects  of  simulator  use,  and  licensed 
software  and  know-how  to  industry. 
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He  is  currently  working  on  new  approaches  for  training  helicopter  flight  deck  operations  and 
investigating  technologies  for  embedded  training  systems  within  operational  equipment.  In  2007, 
the  Air  Force  Association  of  Canada  recognized  the  efforts  that  he  led  on  helicopter  simulation 
as  an  outstanding  contribution  to  military  aviation  in  Canada. 


Harry  O’Neil 

Harry  O’Neil’s  interest  in  technology  and  education  and  training  began  at  Florida  State 
University.  In  1975,  he  was  recruited  to  join  the  Defense  Advanced  Research  Projects  Agency 
(DARPA).  From  1978  to  1985  he  served  in  various  research  roles  at  the  Army  Research  Institute 
for  Behavioral  and  Social  Sciences  (ARI).  He  was  a  member  of  the  Senior  Executive  Service.  In 
the  mid  1980s,  O’Neil  assumed  a  position  as  Full  Professor  of  Educational  Psychology  and 
Technology  at  USC.  He  has  been  a  member  of  the  Army  Science  Board  and  various  Defense 
Science  Board  Task  Eorces.  He  is  a  Eellow  of  the  American  Psychological  Association  and  a 
Certified  Performance  Technologist.  A  prolific  writer,  he  has  recently  co-edited  four  books — 
What  Works  in  Distance  Learning:  Guidelines  (2005),  Web-Based  Learning:  Theory,  Research, 
and  Practice  (2006),  Assessment  of  Problem  Solving  Using  Simulations  (2008),  and  Computer 
Games  and  Team  and  Adult  Learning  (2008). 


Robert  Patterson 
Washington  State  University 


Professor 

Deparment  of  Psychology 
and  Program  in  Neuroscience 
Washington  State  University 
Pullman,  WA  99164-4820 


Research  Scientist  Consultant 
Eink  Simulation  and  Training 
E-3  Communications 
Air  Eorce  Research  Eaboratory 
Mesa,  AZ  85212 


Email:  rpatter@mail.wsu.edu;  Telephone:  (509)  432-3078 

Areas  Of  Expertise 

Cognitive  Science,  Visual  Perception  &  Applied  Vision,  Experimental  Design,  System 
Dynamics  Modeling 

Education 

1976  B.A.  Behavioral  Science,  San  Jose  State  University 

1978  M.A.  Experimental  Psychology,  San  Jose  State  University 

1984  Ph.D.  Experimental  Psychology,  Vanderbilt  University 

1985-7  Postdoctoral  Research  Eellow,  Visual  Neuroscience,  Northwestern  University 

Selected  Publications 
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Patterson,  R.,  Geri,  G.A.,  Dyre,  B.P.,  Akhtar,  S.C.,  Covas,  C.M.,  &  Pierce,  BJ.  (2005).  Altitude 
control  in  simulated  flight  using  3D  objects  and  terrain  texture.  Journal  of  the  Society  for 
Information  Display,  13,  1039-1044. 

Patterson,  R.,  Geri,  G.A.,  Dyre,  B.P.,  Pierce,  B.J.,  Akhtar,  S.C.,  Covas,  C.M.&  Morgan,  W.  (2006). 
Active  heading  control  in  simulated  flight  based  on  vertically  extended  contours.  Perception  & 
Psychophysics,  68,  593-600. 

Patterson,  R.,  Winterbottom,  M.  &  Pierce,  B.J.  (2006).  Perceptual  Issues  with  the  use  of  head- 
mounted  displays  (review  paper).  Human  Factors,  48,  555-573. 

Winterbottom,  M.D.,  Patterson,  R.,  Pierce,  B.J.,  Covas,  C.  &  Winner,  J.  (2007).  Depth  of  focus 
and  visual  recognition  of  imagery  presented  on  simultaneously  viewed  displays:  Implications  for 
head-mounted  displays.  Human  Factors,  49,  907-919. 

Patterson,  R.,  Winterbottom,  M.,  Pierce,  B.J.  &  Fox,  R.  (2007).  Binocular  rivalry  and  head- 
mounted  displays  (review  paper).  Human  Factors,  49,  1083-1096. 

Patterson,  R.  (2007).  Human  factors  of  3D  displays  (review  paper).  Journal  of  the  Society  for 
Information  Display,  15,  861-871. 

Patterson,  R.  &  Silzars,  A.  (2009).  Immersive  stereo  displays,  intuitive  reasoning,  and  cognitive 
engineering.  Journal  of  the  Society  for  Information  Display,  17,  443-448. 

Patterson,  R.,  Fournier,  L.,  Pierce,  B.P.,  Winterbottom,  M.  &  Tripp,  L.  (2009).  System  dynamics 
modeling  of  the  recognition-primed  decision  model.  Journal  of  Cognitive  Engineering  and 
Decision  Making,  in  press. 

Patterson,  R.,  Pierce,  B.P.,  Bell,  H.,  Andrews,  D.  &  Winterbottom,  M.  (2009).  Training  robust 
decision  making  in  immersive  environments  (review  paper).  Journal  of  Cognitive  Engineering 
and  Decision  Making,  in  press. 
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Robert  W.  Proctor 
Purdue  University 


Distinguished  Professor  of  Psychological  Sciences 
Purdue  University 
703  Third  S., 

West  Lafayette,  IN  47907-2081 
Phone:  (765)  494-0784, 

FAX:  (765)  496-1264, 

Email:  proctor@psych.purdue.edu 

Education 

University  of  Texas,  Austin,  Major:  Psychology,  B.A.,  1970 

University  of  Texas,  Arlington,  Major:  Experimental  Psychology,  M.A.,  1972;  Ph.D.,  1975 

Appointments 

Department  of  Psychological  Sciences,  Purdue  University:  Distinguished  Professor  (2008- 
present). 

Department  of  Psychological  Sciences,  Purdue  University:  Professor  (1988  -  2007). 
Department  of  Psychology,  Auburn  University:  Associate  Professor  (1980  -  1988). 
Department  of  Psychology,  Auburn  University:  Assistant  Professor  (1976  -  1980). 
Department  of  Psychology,  Michigan  State  University:  Assistant  Professor  (1975  -  1976) 

Selected  Publications 

Yamaguchi,  M.,  &  Proctor,  R.  W.  (2009).  Transfer  of  learning  in  choice  reactions: 
Contributions  of  specific  and  general  components  of  manual  responses.  Acta  Psychologica,  130, 
1-10. 

Bae,  G.  Y.,  Cho,  Y.  S.,  &  Proctor,  R.  W.  (2009).  Transfer  of  orthogonal  stimulus-response 
mappings  to  an  orthogonal  Simon  task.  Quarterly  Journal  of  Experimental  Psychology,  62, 
746-765. 

Proctor,  R.  W.,  Yamaguchi,  M.,  Zhang,  Y.,  &  Vu,  K.-P.  E.  (2009).  Influence  of  visual  stimulus 
mode  on  transfer  of  acquired  spatial  associations.  Journal  of  Experimental  Psychology: 
Learning,  Memory,  and  Cognition,  35,  434-445. 

Proctor,  R.  W.,  &  Van  Zandt,  T.  (2008).  Human  factors  in  simple  and  complex  systems  (2"‘^  ed.). 
Boca  Raton,  PE:  CRC  Press. 

Proctor,  R.  W.,  Yamaguchi,  M.,  &  Vu,  K.-P.  E.  (2007).  Transfer  of  noncorresponding  spatial 
associations  to  the  auditory  Simon  task.  Journal  of  Experimental  Psychology:  Learning, 
Memory,  and  Cognition,  33,  245-253. 

Wang,  D.-Y.  D.,  Proctor,  R.  W.,  &  Pick,  D.  P.  (2007).  Acquisition  and  transfer  of  attention- 
allocation  strategies  in  a  multiple-task  work  environment.  Human  Eactors,  49,  995-1004. 

Proctor,  R.  W.,  Vu,  K.-P.  E.,  &  Pick,  D.  P.  (2005).  Aging  and  response  selection  in  spatial 
choice  tasks.  Human  Eactors,  47,  250-270. 
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Proctor,  R.  W.,  &  Vu,  K.-P.  L.  (2006).  Stimulus-response  compatibility  principles:  Data,  theory, 
and  application.  Boca  Raton,  FL:  CRC  Press. 

Proctor,  R.  W.,  &  Vu,  K.-P.  L.  (2006).  Laboratory  studies  of  training,  skill  acquisition,  and 
retention  of  performance.  In  K.  A.  Ericsson,  N.  Chamess,  P.  J.  Feltovich,  &  R.  R.  Hoffman 
(Eds.),  Cambridge  handbook  of  expertise  and  expert  performance  (pp.  265-286).  Cambridge, 
UK:  Cambridge  University  Press. 

Proctor,  R.  W.,  Tan,  H.  Z.,  Vu,  K.-P.  E.,  Gray,  R.,  &  Spence,  C.  (2005).  Implications  of 
compatibility  and  cuing  effects  for  multimodal  interfaces.  In  D.  D.  Schmorrow  (Ed.), 
Foundations  of  augmented  cognition  (pp.  3-12).  Mahwah,  NJ:  Eawrence  Erlbaum  Associates. 

Proctor,  R.  W.,  &  Dutta,  A.  (1995).  Skill  acquisition  and  human  performance.  Thousand  Oaks, 
CA:  Sage. 

Editorial  Positions 

Editor:  American  Journal  of  Psychology,  March  I,  2008  to  present;  Editor:  Behavior  Research 
Methods,  Instruments,  &  Computers,  1994-1999;  Associate  Editor:  Memory  &  Cognition,  1986- 
1993;  International  Journal  of  Human-Computer  Interaction,  1992-2000;  Quarterly  Journal  of 
Experimental  Psychology,  2001-2005;  Consulting  Editor:  Perception  &  Psychophysics,  1983- 
1998;  International  Journal  of  Human-Computer  Interaction,  2001-present;  Journal  of 
Experimental  Psychology:  Human  Perception  and  Performance,  2001 -present;  Acta 
Psychologica,  2006-present;  Human  Eactors,  2006-present;  Psychological 
Research/Psychologische  Eorschung,  2009-present;  Memory  &  Cognition,  2009-  present. 


Rand  J.  Spiro 
Michigan  State  University 

Rand  J.  Spiro,  Ph.D.,  is  a  professor  of  Educational  Psychology  and  Educational 
Technology  at  Michigan  State  University  and  a  principal  investigator  in  the  Eiteracy 
Achievement  Research  Center  (EARC).  Before  coming  to  MSU,  Dr.  Spiro  was  a  Distinguished 
Senior  Scholar  in  the  College  of  Education  at  the  University  of  Illinois  at  Urbana-Champaign, 
where  he  was  Professor  of  Educational  Psychology,  Psychology,  and  the  Beckman  Institute  for 
Advanced  Science  and  Technology.  He  has  been  a  Visiting  Scientist  in  Psychology  and 
Computer  Science  at  Yale  University,  where  he  worked  in  the  Yale  Artificial  Intelligence 
Eaboratory,  and  a  Visiting  Professor  of  Education  at  Harvard  University.  He  was  a  founding 
member  and  served  for  a  time  as  co-director  of  the  national  Center  for  the  Study  of  Reading  at 
the  University  of  Illinois.  He  also  served  as  Chair  of  the  Department  of  Educational  Psychology 
at  Illinois. 

Spiro  has  been  quoted  and  his  ideas  discussed  in  such  national  media  outlets  as,  most 
recently,  the  New  York  Times,  the  Washington  Post,  and  the  Wall  Street  Journal.  Altogether, 
Spiro  has  made  over  100  invited  presentations  to  scholarly  groups,  including  major  invited 
addresses  to  professional  societies  such  as  the  American  Psychological  Association  (co- 
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sponsored  by  the  Experimental  Psychology  and  Educational  Psychology  divisions),  the 
Cognitive  Science  Society,  the  American  Educational  Research  Association  (three  invited 
addresses:  one  for  Division  C,  one  for  Division  I  and  one  for  Division  K).  the  International 
Reading  Association,  the  National  Conference  on  Research  in  English  (President's  Invitational 
Address),  the  Semiotics  Society  of  America,  the  National  Reading  Conference  (two  major 
invited  addresses),  a  Director's  Invitational  Seminar  at  the  Beckman  Institute  for  Advanced 
Science  and  Technology,  invited  panelist  at  the  Prank! in  Institute  Awards  (panel  for  Donald 
Norman,  laureate  in  computer  and  cognitive  science),  invited  colloquia  at  universities  such  as 
Harvard,  Yale,  Michigan,  Northwestern,  New  York  University,  Georgia,  Indiana,  Washington, 
Arizona,  Delaware,  Georgia  Tech,  Wisconsin,  McGill,  Eisbon,  Rome,  Tokyo,  Canterbury  (New 
Zealand),  universities  in  Beijing  and  Shanghai,  and  various  universities  in  Portugal  (under  the 
sponsorship  of  the  Joint  Euso-American  Pulbright  Commission),  invited  talks  at  NATO 
conferences  in  Brussels  (Belgium),  Eeuven  (Belgium),  Espinho  (Portugal),  Crete  (Greece),  Aix- 
en-Provence  (Prance),  and  Edinburgh  (UK),  and  numerous  other  invited  presentations  around  the 
world. 


He  has  been  a  principal  or  co-principal  investigator  on  grants  from  the  National  Science 
Poundation,  the  US  Department  of  Education,  the  US  Department  of  Health  and  Human 
Services,  the  Spencer  Poundation,  the  Joyce  Poundation,  IBM,  Apple,  the  Department  of 
Defense,  among  others. 

Dr.  Spiro  is  the  co-originator  of  Cognitive  Plexibility  Theory  and  its  application  to 
innovative  approaches  to  hypermedia  design.  His  research  areas  include  new  forms  of  reading 
and  learning  on  the  Web,  knowledge  acquisition  in  complex  domains,  hypermedia  learning 
environments,  multimedia  case-based  methods  in  professional  education,  experience 
acceleration,  biomedical  cognition,  learning  in  history,  and  constructive  processes  in 
comprehension  and  recall.  Much  of  his  research  is  concerned  with  determining  how  learning 
should  proceed  so  that  tendencies  toward  conceptual  oversimplification  are  counteracted  and  a 
wide  range  of  future  applications  of  knowledge  are  supported.  A  central  focus  is  the 
development  and  testing  of  theory-based  hypermedia  learning  environments  designed  to  promote 
cognitive  flexibility.  His  publications  include  the  books  Schooling  and  the  Acquisition  of 
Knowledge;  Theoretical  Issues  in  Reading  Comprehension;  Cognition,  Education,  and 
Multimedia;  and  Hypertext  &  Cognition,  as  well  as  numerous  articles  in  scholarly  journals  and 
chapters  in  edited  volumes. 
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Kurt  VanLehn 


Arizona  State  University 


Kurt  VanLehn  is  a  Professor  in  the  School  of  Computing  and  Informatics  at  Arizona  State 
University.  He  received  a  Ph.  D.  from  MIT  in  1983  in  Computer  Science,  was  a  post-doc  at  BBN 
and  Xerox  PARC,  joined  the  faculty  of  Carnegie-Mellon  University  in  1985,  moved  to  the 
University  of  Pittsburgh  in  1990  and  joined  ASU  in  2008.  He  founded  and  co-directed  two  large 
NSF  research  centers  (Circle;  the  Pittsburgh  Science  of  Learning  Center).  He  has  published  over 
125  peer-reviewed  publications,  is  a  fellow  in  the  Cognitive  Science  Society,  and  is  on  the 
editorial  boards  of  Cognition  and  Instruction,  and  the  International  Journal  of  Artificial 
Intelligence  in  Education. 

Dr.  VanLehn's  research  focuses  on  applications  of  artificial  intelligence  to  education  and 
cognitive  modeling.  Some  of  his  recent  projects  are:  Andes,  an  intelligent  tutoring  system  for  a 
full  year  of  college/high  school  physics  that  improves  students  grades  by  approximate  a  letter 
grade  and  is  in  daily  use  around  the  country;  Why2-Atlas  and  Cordillera,  two  intelligent  tutoring 
systems  that  pioneered  the  use  of  natural  language  dialogues  for  science  teaching  and  have  been 
shown  to  be  just  as  effective  as  expert  human  tutors;  Pyrenees,  an  intelligent  tutoring  system  that 
successfully  caused  inter-domain  transfer  by  implicitly  teaching  a  meta-cognitive  strategy;  and 
Cascade,  a  highly  accurate  cognitive  model  of  human  students  learning  physics  that  accounts  for 
the  interaction  of  self-explanation  and  analogy. 


Karen  Velkey 

Northrop  Grumman  Corporation 

Education 

Millsaps  College,  Jackson,  MS  BA  1987-1991  Psychology 

The  University  of  Montana,  Missoula,  MT  MBA  1992-1994  Business 

Employment  History 

Compensation  Manager,  Northrop  Grumman  Corporation,  2008-Present 
Compensation  Analyst,  Northrop  Grumman  Corporation,  2001-2008 

HR  Associate,  Payroll  Coordinator,  Purchasing  Agent,  Double  G  Coatings,  Byrum,  Mississippi, 
1998-2001 

Visiting  Instructor,  University  of  Montana,  School  of  Business  Administration,  Missoula,  MT 
1995-1997 

Compensation  &  Benefits,  Sr.  Patrick  Hospital,  Missoula,  MT  1994-1996 

Karen  has  over  ten  years  experience  in  Human  Resources,  with  an  emphasis  in  Compensation. 
Her  work  in  compensation  includes  job  analysis  and  evaluation,  salary  structure  design  and 
implementation,  and  bonus  administration.  She  recently  led  a  2  year  project  for  Northrop 
Grumman  to  standardize  job  coding,  titling,  and  baseline  market  data  across  the  corporation. 
She  continues  to  work  with  functional  groups  across  the  organization  to  align  job  families  and 
titles  with  competency  models  and  organizational  development  goals.  Karen  also  serves  on  the 
High  Technology  Advisory  Committee  for  the  SIRS  (Salary  Information  Retrieval  System) 
survey  group  of  Organization  Resource  Consultants.  Karen  taught  Organizational  Behavior  and 
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Personal  Selling  at  the  University  of  Montana,  and  has  developed  and  provided  training  in  all 
aspects  of  HR  including  job  evaluation,  market  data  analysis,  compliance,  safety,  and  customer 
service. 


Colonel  Daniel  R.  Walker 
Air  Force  Research  Laboratory 

Colonel  Daniel  R.  Walker  is  the  Chief,  Warfighter  Readiness  Research  Division,  and 
Commander,  Mesa  Research  Site,  Air  Force  Research  Laboratory,  Mesa,  Arizona.  He  directs 
research  and  development  (R&D)  for  Air  Force  Material  Command’s  premier  training  research 
organization,  developing  advanced  systems,  tools,  &  methods  to  improve  warfighter 
performance.  Col  Walker  manages  over  $50M  annually  to  facilitate  programs  ensuring  optimum 
solutions  to  real  world  training  requirements,  integrating  efforts  between  warfighters,  force 
providers,  combatant  commands,  industry,  academia,  and  acquisition  agencies.  The  division 
employs  over  250  uniformed  military,  government  civilian  and  contractor  scientists,  engineers 
and  technicians.  He  directs  all  civil  engineering,  security,  morale  and  welfare,  and  professional 
development  activities  for  the  site. 

Col  Walker  holds  four  Master’s  degrees  and  is  pursuing  a  PhD  in  Training  and  Performance 
Improvement.  He  has  extensive  experience  in  aircrew  training  and  joint  operations  including 
command  of  the  B-1  Division,  USAF  Weapons  School,  and  duty  as  Assistant  Deputy  Director  of 
Operations  for  JTF  SWA,  where  he  participated  in  the  planning  and  execution  of  Operation 
Desert  Fox,  the  first  combat  employment  of  the  B-1. 

Education 

2002  M.A.,  National  Security  and  Strategic  Studies,  Naval  War  College,  Newport,  Rhode  Island 
2000  Air  War  College 

1996  Joint  and  Combined  Staff  Officer  School,  Armed  Forces  Staff  College,  Norfolk,  VA 
1995  M.A.A.S,  Airpower  Arts  and  Science,  School  of  Advanced  Airpower  Studies,  Maxwell  AFB,  AL 
1994  M.M.A.S,  Military  Arts  and  Science,  US  Army  Command  and  General  Staff  College,  Ft 
Leavenworth,  KS 

1992  USAF  Weapons  School,  Nellis  AFB,  NV  and  Ellsworth  AFB,  SD 
1985  M.A.,  Management  and  Supervision,  Central  Michigan  University 
1985  Marine  Corps  Command  and  Staff  College 
1985  Squadron  Officers  School,  Maxwell  AFB  AL 
1980  B.S.,  Biology,  United  States  Air  Force  Academy 


Assignments 

May  05-Present 

Jun  04-May  05 
Jun  02-Jun  04 
Jun  01-Jun  02 
Jun  99-Jun  01 

Jul  98-May  99 


Chief,  Warfighter  Readiness  Research  Division  and 

Commander,  Mesa  Research  Site,  Air  Force  Research  Laboratory,  Mesa,  AZ 
Chief,  Nuclear  Policy  Division,  The  Joint  Staff,  J-5,  Washington,  D.C. 

Chief,  Strategic  Nuclear  Policy,  The  Joint  Staff,  J-5,  Washington,  D.C. 

Naval  War  College,  Newport,  RI 

Commander,  USAF  Weapons  School  B-1  Division  and  Detachment  4,  57*  Wing, 
Ellsworth  AEB,  SD. 

Assistant  Operations  Officer,  37*  Bomb  Squadron,  Ellsworth  AEB,  SD 
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Jul  95-Jun  98 
Jun  94-Jun  95 
Jun  93-Jun  94 
Aug  92-Jun  93 
Feb  92-Aug  92 
Jun  91— Feb  92 
Mar  91-Jun  91 
Nov  89-Mar  91 
Nov  88-Nov  89 
Aug  87-Nov  88 
Oct  86-Aug  87 
Jun  85-Oct  86 
Jun  82-Jun  85 

Jul  80-Jun  82 


Chief,  Air  Operations,  US  Atlantic  Command,  J-3,  Norfolk,  VA 

School  of  Advanced  Airpower  Studies,  Maxwell  AFB,  AL 

US  Army  Command  and  General  Staff  College,  Ft  Leavenworth,  KS 

Operations  Officer,  B-1  Division,  USAF  Weapons  School,  Ellsworth  AFB,  SD 

Chief,  Flight  Operations,  Bomber/Tanker  Employment  School,  Ellsworth  AEB,  SD 

Executive  Officer,  Strategic  Warfare  Center,  Ellsworth  AEB,  SD 

B-1  Tactics  Program  Manager,  99*  Strategic  Weapons  Wing,  Ellsworth  AEB,  SD 

Chief  of  Offensive  Systems,  25*  Strategic  Training  Squadron,  Ellsworth  AEB,  SD 

Chief,  Target  Study  Section,  28  *  Bomb  Wing,  Ellsworth  AEB,  SD 

B-1  Instructor  Offensive  Systems  Officer,  37*  Bomb  Sq,  Ellsworth  AEB,  SD 

B-1  Combat  Crew  Training,  338*  Strategic  Training  Sq,  Dyess  AEB,  TX 

Target  Study  Officer,  319*  Bomb  Wing  Grand  Eorks  AEB,  ND 

B-52G/H  Evaluator  Navigator,  Instructor  Radar  Navigator,  46*  Bomb  Sq,  Grand 

Eorks  AEB,  ND 

Undergraduate,  Advanced  Elying,  and  B-52  Combat  Crew  Training 


Flight  Information 

Rating:  Master  Navigator 

Elight  Hours:  More  than  2,500 

Aircraft  Elown :  B  - 1 ,  B  -  5  2 


Awards 

Defense  Superior  Service  Medal 

Joint  Meritorious  Service  Medal 

Meritorious  Service  Medal  with  two  oak  leaf  clusters 

Air  Eorce  Commendation  Medal 

Joint  Service  Achievement  Medal  with  oak  leaf  cluster 

Air  Eorce  Achievement  Medal  with  oak  leaf  cluster 

Combat  Readiness  Medal  with  oak  leaf  cluster 

National  Defense  Service  Medal  with  gold  star 

Armed  Eorces  Expeditionary  Medal 

War  on  Terror  Service  Medal 


Other  Achievements 

1989  Eairchild  Trophy  for  Most  Outstanding  Bombing  and  Navigation  in  Strategic  Air 
Command 

Professional  Certifications 

Defense  Acquisition  Workforce  Improvement  Act:  Systems  Planning,  Research,  Development 
and  Engineering,  Eevel  III 
Program  Management,  Eevel  I 

Selected  Publications 

Walker,  D.  R.,  &  Geiss,  K.  T.  (In  Press).  Euture  Air  Eorce  Training.  In  D.  Schmorrow,  J.  Cohn  & 
D.  Nicholson  (Eds.),  The  PSI  Handbook  of  Virtual  Environments  for  Training  and 
Education:  Developments  for  the  Military  and  Beyond.  Westport,  CT:  Praeger  Security 
International. 


284 


Walker,  D.  R.  (1996).  The  Organization  and  Training  of  Joint  Task  Forces.  Maxwell  AFB,  AL: 
Air  University  Press. 

Professional  Associations 

Air  Force  Association 

Armed  Forces  Chapter-  International  Society  for  Performance  Improvement 
American  Society  for  Training  and  Development 

Association  of  Old  Crows  -  The  Electronic  Warfare  and  Information  Operations  Association 

Kappa  Delta  Pi  -  Honor  Society  for  Education 

National  Air  and  Space  Society 

National  Training  and  Simulation  Association 

Effective  Dates  Of  Promotion 

Colonel  01  Nov  2002 

Eieutenant  Colonel  01  July  1997 

Major  01  June  1992 

Captain  28  May  1984 

Eirst  Eieutenant  28  May  1982 

Second  Eieutenant  28  May  1980 

(Current  as  of  Sep  2008) 


David  M.  Ziebell 
Electric  Power  Research  Institute 

David  Ziebell  is  a  Senior  Project  Manager  in  the  Equipment  Reliability  program  area  of  the 
Nuclear  Sector.  His  recent  research  activities  focus  on  Expert  Knowledge  Capture  and  Transfer, 
one  of  several  industry  responses  to  knowledge  loss  due  the  aging  workforce.  Ongoing  research 
and  development  activities  support  nuclear  generating  plant  operations  and  maintenance  with 
organizational  performance  and  process  improvement,  and  procedure  and  work  practice 
effectiveness. 

Mr.  Ziebell  joined  EPRI  in  2003  as  the  Manager,  Human  Performance  Technology.  His  activities 
focused  on  development  of  leading  indicators  of  human  performance,  supporting  performance 
improvement  process  benchmarking  with  the  Nuclear  Energy  Institute,  and  developing  cross¬ 
functional  aspects  of  human  performance  for  all  sectors  in  the  electric  industry. 

Before  joining  EPRI,  Mr.  Ziebell  worked  at  the  Institute  of  Nuclear  Power  Operations,  and  was 
involved  in  various  activities  providing  evaluation  and  assistance  services  to  nuclear  utilities  in 
both  operations  and  human  performance.  He  also  worked  at  Southern  California  Edison  as  senior 
reactor  operator  and  shift  manager,  and  served  as  a  submarine  qualified  officer  in  the  U.S.  Navy. 

Mr.  Ziebell  received  a  BS  degree  in  Atmospheric  Science  from  Purdue  University. 
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Appendix  D 


Pre-Meeting  Position  Papers 


Neil  Chamess,  Ph.D.,  Florida  State  University 
Jared  Freeman,  Ph.D.,  Aptima,  Inc. 

Andy  Gibbons,  Ph.D.,  Brigham  Young  University 

Frank  L.  Greitzer,  Ph.D.,  Pacific  Northwest  National  Laboratory 

Alice  F.  Healy,  Ph.D.,  University  of  Colorado 

Robert  Hubal,  RTI  International 

Harry  O'Neil,  Ph.D..  University  of  Southern  California 

Robert  Patterson,  Ph.D.,  Washington  State  University 

Robert  Proctor,  Ph.D..  Purdue  University 

Karen  Velkey,  Northrop-Grumman 

David  Ziebell,  Electric  Power  Research  Institute 
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Neil  Charness,  Ph.D. 

Florida  State  University 

I’m  going  to  adopt  the  strengths,  weaknesses,  opportunities,  and  threats  (SWOT)  framework  for 
musing  about  some  of  the  topisc. 

Accelerated  Learning  Topic 

Strengths:  We  are  beginning  to  understand  what  are  effective  schedules  for  training  for  retention 
(e.g.,  for  foreign  language  -  Pavlik  &  Anderson).  We  have  very  sophisticated  tools  for 
analyzing  developmental  studies  (e.g.,  longitudinal  data  sets)  such  as  multi-level  modeling 
techniques,  trajectory  analysis  techniques. 

Weaknesses:  Most  lab  studies  of  learning  involve  declarative  memory  skills  (word  list  and  paired 
associate  tasks)  rather  than  procedural  skills  and  the  latter  may  be  more  relevant  for  military 
tasks  such  as  piloting,  marksmanship,  mine  detection,  which  involve  a  mix  of  both  types  of 
learning. 

Opportunities:  Developing  and  comparing  different  training  techniques  (hands-on,  simulation 
environments  of  varying  fidelity)  and  doing  longitudinal  studies  within  a  diverse  population  of 
learners. 

Threats:  Not  having  validated  measurement  tools  for  proficiency  assessment  for  many  real-world 
tasks,  generalizing  to  diverse  populations  of  trainees  (diversity  of  educational  backgrounds,  ages, 
and  proficiency  levels  after  initial  training).  Ability  to  mount  long  enough  longitudinal  studies  to 
model  typical  interrupted  careers. 

Retention  of  Skill  Topic 

Strengths:  We  know  a  great  deal  about  memory  for  information  post  learning,  ranging  from 
minutes  (lab  science)  to  years  (Open  University  psychology  course  material)  to  decades 
(language  skills  -  Bahrick). 

Weaknesses:  We  do  not  have  much  information  about  individual  difference  predictors  of  skill 
retention,  such  as  age,  gender,  cognitive  ability  level,  education  level,  etc.  We  lack  information 
about  how  booster  sessions  (number,  spacing)  affect  learning  and  forgetting  rates.  Do  not  have 
solid  data  about  job  requirement  changes  and  transfer  of  skills  within  careers  (e.g.,  piloting  a  jet 
versus  a  UAV;  Air  Traffic  Control  with  changing  technology)  and  knowledge  obsolescence  in 
jobs  and  what  motivates  people  to  engage  in  the  type  of  deliberate  practice  needed  to  maintain 
and  develop  skill. 

Opportunities:  Tracking  job  requirements  longitudinally  in  a  rapidly  changing  job  environment 
(e.g.,  technology  adoption  issues);  understanding  career  trajectories  in  the  military  and  how  that 
affects  recruitment  and  retention,  particularly  within  an  aging  civilian  population;  developing 
measures  of  motivation  that  might  predict  attrition. 

Threats:  Same  as  above  for  accelerated  learning. 
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Jared  Freeman,  Ph.D. 
Aptima,  Inc. 


The  Accelerated  Learning  workshop  will  address  a  variety  of  fundamental  issues  in  the  learning 
sciences  (e.g.,  measurement,  scaffolding,  resilience). 

We  posit  that  instructional  technology  should  be  considered  early  in  these  discussions,  both 
because  the  planned  topics  may  inform  the  definition  of  requirements  for  such  technology,  and 
because  appropriate  technology  potentially  enhances  our  ability  to  address  the  workshop  issues 
over  the  long  term. 

Several  classes  of  technology  may  be  central  to  implementing  Accelerated  Learning  concepts: 

•  Pervasive  technology  for  human  performance  measurement  is  needed  to  estimate  the  effects  of 
training,  in  the  short  term,  and  to  validate  and  calibrate  the  impact  of  competencies  on  task 
performance,  in  the  long  term. 

•  Scenario  engineering  technology  is  required  to  dynamically  adapt  training  simulations  and 
games  in  ways  that  ensure  students  are  continuously  stimulated  by  events  that  exercise  and 
extend  their  competencies.  In  the  long  term,  such  technology  should  help  us  to  assess  whether 
training  in  specific  competencies  is  feasible  in  simulations. 

•  Formal  (computational)  representations  of  competency  and  its  impact  on  operational  tasks  (e.g., 
accelerating  performance,  raising  quality)  are  needed  to  model  and  support  the  performance  of 
military  organizations  under  different  assumptions  about  staff  composition  and  size. 
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Andy  Gibbons,  Ph.D. 

Brigham  Young  University 

The  team  aspect  of  performance  is  one  of  the  most  critical  issues.  Therefore,  consideration  must 
be  given  to: 

a.  Helping  team  members  become  metacognitive  about  their  participation  in  a  team. 

b.  Allowing  practice  with  feedback  to  occur  at  the  team  level  as  frequently  as  required  for 
expertise-building  in  that  set  of  skills.  This  will  require  a  socially-networked  venue  among 
widely-distributed  personnel. 

c.  Giving  team  members  a  clear  idea  of  the  “personas” — the  multiple  contributory  functions — 
that  make  up  a  full-function  team.  In  this  respect,  there  are  useful  clues  in  The  Ten  Faces  of 
Innovation,  by  Kelley.  The  book  is  written  with  design  teams  in  mind;  the  priority  on  training 
and  performance  teams  that  can  innovate  and  solve  problems  fits  nicely  with  this  theme. 

Several  workshop  issues  such  as  deep  structure,  fine  discriminations,  transfer,  progressive 
development,  and  the  learning  of  tacit  knowledge  that  is  useful  in  real  situations  can  be  helped 
by  a  re-visitation  of  the  “increasingly  complex  microworlds”  theme  as  expressed  by  Burton, 
Brown,  and  Fischer  (Burton,  R.  R.,  Brown,  J.  S.,  &  Fischer,  G.  (1984):  “Skiing  as  a  Model  of 
Instruction”  —  In  B.  Rogoff,  &  J.  Lave  (Eds.),  Everyday  Cognition:  Its  Development  in  Social 
Context,  Harvard.  Access  at:  http://13d.cs.colorado.edu/~gerhard/papers/skiing-paper-1984.pdf). 
There  is  a  precision  attainable  in  training  effect  through  careful  selection  and  sequencing  of 
training  problems.  There  are  efficiency,  contrast,  and  repetition  effects  possible  when  staccato 
problem  sequences  are  used  which  change  only  one  or  a  few  problem  variables  of  interest. 
University  Press,  Cambridge,  MA  and  Eondon,  pp.  139-150.).  When  ICM  was  originally 
expressed  by  them  (and  later  by  others),  it  was  done  qualitatively.  Perhaps  the  time  has  come  to 
give  a  more  quantitative  treatment  to  this  concept  which  would  allow  the  scaling  of  problems 
and  more  nuanced  sequencing  schemes.  Work  by  Victor  Bunderson  on  “domain  theory”  is 
relevant  here.  (See  Bunderson,  C.V.  (2006).  Developing  a  domain  theory:  Defining  and 
exemplifying  a  learning  theory  of  progressive  attainments.  In  M.  Gamer,  G.  Engelhard,  M. 
Wilson,  &  W.  Eisher  (Eds.),  Advances  in  Rasch  measurement.  Volume  I.  Maple  Grove,  MN: 
JAM  Press.) 

It  seems  relevant  to  bring  up  the  design-based  research  approach  in  connection  with  this 
ambitious  research  agenda.  DBR  is  appropriate  for  exactly  the  messy  and  complex  problems  this 
workshop  is  dealing  with.  In  principle,  it  would  allow  a  more  flexible  approach  to  both  problem¬ 
finding  and  solution  framing,  and  it  pre-supposes  inter-silo,  team-based,  and  exploratory  research 
processes. 

It  may  be  appropriate  to  consider  the  area  of  weather  forecasting  and  analysis  as  one  of  the 
subject-matter  areas  to  research.  Several  factors  recommend  this,  including  its  importance  to  all 
of  the  military,  its  complexity,  its  inherently  team-based  nature,  and  an  ongoing  test  bed  of 
instmctional  design  and  development  that  exists  in  the  COMET  Project  of  the  University 
Corporation  for  Atmospheric  Research  at  Boulder,  Colorado.  This  project  has  developed  a 
massive  body  of  training  which  is  used  by  a  very  large  international  audience.  It  represents  a 
large  and  growing  design/development  activity  that  likes  to  partner  with  special  needs.  Among 
the  project’s  sponsors  already  are  the  Army,  Navy,  Air  Eorce,  and  Marines.  The  project  would 
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provide  an  excellent  collaborative  partner  which  is  able  to  supply  a  very  large  number  of 
participants  who  represent  an  appropriate  target  population  and  a  ready-made  distribution  system 
for  trying  out  and  iterating  research  products. 

Finally,  I  recommend  the  value  of  a  little-known  idea  which  I  will  call  “Spotting  Operational 
Principles”  as  a  theme  within  the  research.  The  notion  of  an  operational  principle  was  introduced 
by  Michael  Polanyi  (Polanyi,  M.  (1958).  Personal  Knowledge.  University  of  Chicago  Press).  It 
focuses  the  mind  on  the  core  essential  mechanisms  by  which  an  artifact  works.  Too  often  in  our 
training  we  conduct  the  learner  through  a  forest  of  details  before  acquainting  him/her  with 
abstract  inner  mechanisms.  Learners  are  thus  not  trained  to  drive  to  the  heart  of  their  knowledge 
and  see  learning  as  the  amassing  of  bulk  rather  than  incisive  insight  into  a  handful  of  defining 
principles.  Teaching  learners  to  search  for  operational  principles  will  accelerate  learning  and 
organize  what  is  learned  for  long-term  retention  and  independent  reasoning  and  knowledge- 
extension. 
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Frank  L.  Greitzer,  Ph.D. 

Pacific  Northwest  National  Laboratory 


Key  Issues 

•  Defining  the  ingredients  of  expertise  for  a  problem  domain 

0  Breaking  down  expert  knowledge  into  critical  concepts:  mental  models, 
associated  cues/patterns  (within  a  cognitive  engineering/naturalistic  decision 
making  approach) 

0  Defining  conceptual  elements  that  reflect  varying  levels  of  expertise;  also 
misconceptions  and  characteristics  of  less  experienced  learners 

0  The  expert  knowledge  (ingredients  of  expertise)  needs  to  be  represented  in  a 
computational  form.  Formal  ontological  languages  and  knowledge-bases  to 
represent  the  domain,  data,  and  its  axioms  are  important  for  reasoning  within  the 
domain. 

•  Developing  measures/metrics 

0  Identify  operative  mental  models 

0  Assess  expertise  -  using  the  knowledge  base. 

0  Identify  need  for  mitigation  -  recognition/inference  of  the  distance  between  the 
current  mental  model  and  an  optimal  mental  model  can  be  done  based  on 
knowledge  representation  methods/tools;  that  distance  and  differences  in  patterns 
may  be  used  to  develop  or  apply  strategies  for  mitigation  using  a  different  mental 
model. 

Pertinent  Theories 


•  Expertise/Knowledge 

0  Naturalistic  Decision  Making — Recognition  Primed  Decision  Making 

•  Metacognition  concepts — needed  to  support  insight/understanding  about  misconception 
or  being  on  wrong  track,  needing  to  switch  to  another  mental  model  for  example 

•  Experiential  learning 

0  Guided  Discovery  Eearning 
0  Problem-based  learning 
0  Adaptive/Predictive  case-based  learning 

Research  Challenges 

•  Development  of  sufficiently  accurate  and  detailed  knowledge  representations  to  support 
automated  reasoning/intelligent  training  management 

•  Knowledge  representation  should  reflect  mental  models 

0  Mental  models  should  be  developed  that  correspond  to  different  levels  of 
expertise 
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0  Mental  models  also  should  be  defined  to  represent  expected  misconceptions  and 
reasoning  failures  of  less  experienced  trainees 


•  How  to  recognize  performance  associated  with  a  given  level  of  expertise  and  mental 
model:  A  key  requirement  for  accelerating  learning  is  to 

0  Recognize  mental  model  being  employed  in  solving  a  problem  (associated  with  a 
learning  state) 

0  Identify  observable  performance  “indicators”  associated  with  mental  models 

•  Apply  appropriate  training  management  action  based  on  identified  learning  state 

0  Mitigation  (e.g.,  hints,  scaffolding) 

0  Adaptable  scenarios  to  increase  or  decrease  complexity  based  on  learning  state 
Opportunities 

•  Among  numerous  examples  of  domains  that  require  years  of  “on  the  job”  experience,  the 
electric  power  industry  (generation  and  transmission)  is  an  excellent  potential  application 
domain  for  R&D  on  approaches  to  accelerate  expertise  acquisition. 

•  Major  challenge  for  training  in  this  industry. . .  aging  workforce,  gap  between  seasoned 
experts  and  apprentice/journeyman  operators,  cultural  and  background/experience 
differences  ,  limited  funding  for  training. 
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Alice  F.  Healy,  Ph.D. 

University  of  Colorado 

Optimal  training  should  be  efficient,  durable,  and  flexible.  On  the  basis  of  prior  research,  we 
have  developed  a  set  of  training  principles  that  optimize  the  speed  of  training,  the  long-term 
retention  of  trained  knowledge  and  skills,  and  the  transfer  of  trained  performance  to  new 
circumstances.  The  following  is  a  brief  summary  of  some  illustrative  training  principles. 

(1)  Increasing  interference  during  training  has  been  shown  to  impede  training  efficiency  but 
ultimately  to  enhance  the  durability  and  flexibility  of  what  is  learned.  By  the  contextual 
interference  principle,  interference  during  learning  facilitates  later  retention  and  transfer  (Battig, 
1979;  Schneider,  Healy,  Ericsson,  &  Bourne,  1995). 

(2)  It  has  been  shown  that  many  things  that  make  learning  difficult  facilitate  transfer  to  a  new 
task  as  well  as  long-term  retention  of  the  original  task.  By  the  training  difficulty  principle,  any 
condition  that  causes  a  desirable  difficulty  during  learning  facilitates  later  retention  and  transfer 
(Bjork,  1994;  McDaniel  &  Einstein,  2005;  Schneider,  Healy,  &  Bourne,  2002). 

(3)  Eor  optimal  performance,  the  entire  configuration  of  task  requirements  during  training  needs 
to  match  those  in  the  field  as  closely  as  practically  possible.  By  the  specificity  of  training 
principle,  retention  and  transfer  are  depressed  when  conditions  of  learning  differ  from  those 
during  subsequent  testing  (Healy,  Wohldmann,  Sutton,  &  Bourne,  2006). 

(4)  If  secondary  task  requirements  exist  in  the  field,  then  such  requirement  should  be 
incorporated  into  training  to  provide  optimal  transfer  to  field  performance.  Eikewise,  training 
with  extraneous  secondary  task  requirements  should  not  be  use  if  field  performance  does  not 
include  those  requirements.  By  the  functional  task  principle,  secondary  task  requirements  are 
often  integrated  with  primary  task  requirements  during  learning,  resulting  in  the  acquisition  of  a 
single  functional  task  rather  than  two  separate  tasks  (Healy,  Wohldmann,  Parker,  &  Bourne, 
2005;  Hsiao  &  Reber,  2001). 

(5)  Procedural  information  (knowing  how  to  do  something)  is  more  durable  than  declarative 
information  (knowing  that  something  is  the  case).  However,  durable  performance  lacks 
generality  because  performance  at  test  is  optimal  only  when  the  procedures  acquired  during 
training  are  duplicated  during  testing.  By  the  procedural  reinstatement  principle,  specificity 
(limited  transfer)  occurs  for  tasks  based  primarily  on  procedural  information,  or  skill,  whereas 
generality  (robust  transfer)  occurs  for  tasks  based  primarily  on  declarative  information,  or  facts. 
Thus,  for  skill  learning,  retention  is  strong  but  transfer  is  limited,  whereas  for  fact  learning, 
retention  is  poor  but  transfer  is  robust  (Healy,  2007). 

(6)  Pre-existing  knowledge  can  be  used  to  aid  recall  of  new  unrelated  facts  even  when  the  facts 
themselves  fall  outside  the  domain  of  previous  knowledge.  By  the  strategic-use-of-knowledge 
principle,  learning  and  memory  are  facilitated  whenever  pre-existing  knowledge  can  be 
employed  as  a  mediator  in  the  process  of  acquisition  (Kole  &  Healy,  2007;  Van  Overschelde  & 
Healy,  2001). 
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(7)  Prolonged  work  on  a  given  task  often  results  in  deterioration  of  performance,  despite  ongoing 
skill  acquisition.  Specifically,  prolonged  work  sometimes  produces  an  increasing  speed-accuracy 
tradeoff  in  performance,  such  that  accuracy  declines  over  trials  while  at  the  same  time  response 
speed  improves  (Healy,  Kole,  Buck-Gengler,  &  Bourne,  2004).  The  deterioration  seems  to  be 
attributable  to  task  disengagement  on  the  part  of  subjects.  By  the  cognitive  antidote  principle,  the 
introduction  of  cognitive  activities  can  counteract  task  disengagement  effects,  resulting  in 
performance  maintenance  or  even  improvement  during  sessions  of  prolonged  work  (Kole,  Healy, 
&  Bourne,  2008). 

(8)  Mental  practice  can  serve  as  an  effective  substitute  for  physical  practice  for  both  enhancing 
knowledge  of  particular  sequences  and  improving  general  skill.  Indeed  mental  practice  might  be 
superior  to  physical  practice  under  circumstances  that  promote  retroactive  interference.  By  the 
mental  practice  principle,  mental  practice  can  retard  forgetting  and  promote  transfer  of  training 
to  a  larger  extent  than  can  physical  practice,  which  suffers  from  motoric  interference 
(Wohldmann,  Healy,  &  Bourne,  2007,  2008). 
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Robert  Hubal 
RTI  International 


Main  focus  is  on  intelligently  implemented  simulation-based  training/assessment  systems  that 
present  structured  scenarios  (cases). 

Systems  need  to: 

•  Mimic  the  operational  context. 

•  Precisely  monitor  user  actions  —  map  against  pre-understood  performance  measures  during 
critical  tasks. 

•  Be  parameterized,  so  as  to  be  able  to  dynamically  construct  scenarios: 

0  Baseline  scenarios  for  initial  assessment  against  learning  objectives. 

0  Common  and  rare  scenarios  for  learning  generalized  and  specialized  knowledge  and 
procedures. 

0  Instructional  tailoring,  the  introduction  of  “twists”  into  scenarios  to  address  proximal 
learning  objectives  and  biased  reasoning  —  dependent  on  performance  within  recently- 
completed  scenarios. 

Systems  address  skills: 

•  Skills  proficiency  is  attained  through  scenario-based  acquisition,  practice,  validation. 

0  Proficiency  can  be  attained  with  good  training  design  and  situated  learning  in  a  short  time. 

0  As  distinguished  from  expertise  that  involves  both  proficiency  and  strategic  knowledge  and 
adaptability,  and  typically  requires  more  time  to  achieve. 

•  This  pertains  to  a  range  of  skills: 

0  Hands-on,  well-defined  (e.g.,  operation  of  C2  systems). 

0  Predictive,  less  well-defined  (e.g.,  weather  forecasting). 

0  Interactive,  often  poorly-defined  (e.g.,  human-human  interaction). 

•  Skills  retention  is  achievable  through  the  same  sequence  of  acquisition,  practice,  validation, 
though  with  fewer  scenarios  needed. 

Systems  are  not  single-technology. 

•  Use  a  mix  of  training/assessment  environments. 

0  Acquisition  and  much  practice  usually  most  cost-effective  in  a  classroom  and/or  virtual 
environment. 

0  Advanced  practice  and  some  validation  usually  most  cost-effective  in  a  constructive 
environment. 

0  Final  validation  usually  demands  a  live  or  near-live  environment. 

•  Note:  Validation  used  here  as  a  term  indicating  the  final  step  in  demonstrating  skills 
proficiency.  Assessment  used  here  as  a  test  of  skills  performance. 

0  It  is  important  that  assessment  of  skills  occurs  is  contextually-appropriate.  It  requires 
demonstration  of  skills  in  a  reliable,  valid  scenario. 


296 


Harry  O'Neil,  Ph.D. 
University  of  Southern  California 


Conceptual  Issues 

•  Retention  vs.  transfer  tradeoff 

0  Different  instructional  strategies  and  different  assessments 
0  Transfer  has  multiple  meanings 

0  Testing  knowledge  that  has  not  been  explicitly  taught  (Mayer)  vs.  testing 
knowledge  that  transfers  to  job  setting  (Kirkpatrick). 

•  To  accelerate  learning  with  same  level  of  retention 

0  ISD  (teach  less)  results  in  approximately  30%  reduction  in  time. 

0  SD  plus  technology  results  in  an  additive  percentage  reduction  (x%). 

0  Tailor  by  prior  knowledge  people  who  have  knowledge  are  not  taught  it. 

0  Leverage  motivation  to  accelerate  learning 

•  To  accelerate  learning  with  same  level  of  transfer 

0  Trade  off  retention  (it  will  get  worse) 

0  Manipulate  instruction  (e.g.  varied  examples)  with  more  time. 

•  Skill  decay 

0  Declarative  knowledge  is  less  often  forgotten 
0  Procedural  knowledge  is  more  often  forgotten 
0  Initial  acquisition  is  best  predictor  of  skill  decay 

•  Feedback  matters.  Four  characteristics  of  feedback 

0  Complexity  of  feedback 

■  What  information  is  contained  in  the  feedback  messages 

■  Degree  of  elaboration 
0  Timing  of  feedback 

■  When  is  the  feedback  given  to  students 

■  In  complex  learning  delayed  better  than  immediate 
0  Representation  of  feedback 

■  The  form  of  the  feedback  presented  (text  vs.  graphics) 

0  Frequency  of  Feedback 

0  Slows  down  the  Acquisition  of  Knowledge  but  facilitates  Transfer  of  Knowledge 
Create  Just  In  Time  Learning 

0  Short  slices  of  instruction  (<  5min)  targeted  to  a  single  concept 
0  Delivered  online  using  video  with  narration 
0  Represent  the  domain  using  an  ontology 

0  Estimate  what  the  learner  knows  about  a  domain  given  performance  data  on  assessments 
and  what  is  forgotten 

0  Use  Bayesian  networks  to  fuse  assessment  data  and  infer  understanding  of  domain  and 
topics  within  domain 


297 


0  Need  Just  In  Time  assessments  for  Just  In  Time  learning 

Another  Alternative  for  Accelerated  Learning 

0  Teach  people  to  be  smarter 

0  Goal  is  to  improve  cognitive  and  affective  skills  of  personnel 

0  Rationale 

0  Effective  training  is  a  product  of  quality  of  instruction  and  trainees  with  effective 
learning  strategies 

0  Many  trainees  lack  learning  strategies 

Measurement  Challenges 

0  Performance  assessment 

0  Lack  of  psychometrics  for  Simulation/games 

0  Questions 

0  How  is  difficulty  estimated? 

0  How  many  tasks  are  needed? 

0  Live  with  the  reality  that  performance  assessment  and  time  constraints  lead  to  few  tasks. 
Conduct  generalizability  studies. 

0  What  is  role  of  gender,  ethnicity,  and  prior  experiences? 

0  Best  strategy  to  develop  for  useful,  valid,  fair  and  feasible  measures? 

0  Should  constructs  be  reported  as  single  vs.  multiple  scores  as  profile  and/or  levels? 

Research  Recommendations 

0  Create  test  beds  for  cost-effectiveness  studies 

0  Augmented  cognitive  load  theory  with  improved  measurement  and  motivation 
0  John  Sweller-UNSW 

0  Research  for  teams  accelerated  learning 

0  Create  software  pipeline  from  Basic  Training  to  A-Schools  to  C-Schools  to  fleet 
0  Diagnostic/prescriptive  system 

0  Just  in  time  instruction  and  just  in  time  and  right  sized  assessment  in  context  of  training 
pipelines. 

Cognitive  Load  Theory 

■  Intrinsic  cognitive  load:  depends  on  the  extent  of  element  interactivity  of  the  materials  or 
tasks  that  must  be  learned  (van  Merrienboer  &  Sweller,  2005;  Paas,  Renkl,  &  Sweller, 
2003) 

■  Germane  cognitive  load:  associated  with  processes  a  learner  engages  in  that  are  beneficial 
for  learning  (Sweller,  van  Merrienboer,  &  Paas,  1998) 

■  Extraneous  cognitive  load:  caused  by  mental  activities  during  learning  that  are 
unnecessary  and  so  interfere  with  schema  acquisition  and  automation  (Paas,  Renkl,  & 
Sweller,  2003) 

■  Learning  material  should  be  designed  to  minimize  extraneous  cognitive  load  and 
optimize  germane  cognitive  load  (Cooper,1998;  Sweller,  van  Merrienboer,  &  Paas, 

1998). 
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Robert  Patterson,  Ph.D. 
Washington  State  University 


(1)  The  cognitive  psychology  literature,  in  my  view,  is  convincing  in  showing  that  humans 
possess  a  capacity  for  learning  and  acquiring  knowledge  about  the  statistical  structure  of 
dynamic  events  and  episodes  as  they  go  about  their  lives— i.e.,  the  conditional  probabilities,  joint 
probabilities,  and  raw  frequencies  of  temporal  patterns  across  dynamic  scenes. 

(2)  This  learning  can  be  'unsupervised'  and  passive  in  the  sense  that  individuals  learn  such 
dynamic  statistical  structures  in  the  absence  of  an  intention  to  learn,  in  the  absence  of 
instructions  as  to  where  to  direct  attention,  and  in  the  absence  of  feedback  from  their 
environment. 

(3)  Although  this  issue  is  controversial,  it  does  seem  to  me,  and  many  other  researchers,  that 
much  of  this  information  is  implicit  and  tacit  in  the  sense  that  it  is  very  difficult,  if  not 
impossible,  to  verbalize  fully. 

(4)  The  acquisition  of  early  language  by  human  infants  is  likely  to  be  based  on  this  process. 

(5)  This  process  is  likely  to  be  formally  equivalent,  if  not  the  same,  as  the  processes  underlying 
associative  learning  (e.g.,  Pavlovian  and  operant  conditioning). 

(6)  This  kind  of  implicit  learning  is  very  likely  to  be  one  of  the  foundational  processes 
underlying  intuitive  decision  making  (as  discussed  by  Gary  Klein  and  others  in  the  literature). 

Thus,  my  question  is:  Is  this  process  of  largely  implicit  learning  of  spatio-temporal  statistical 
structures  occurring,  in  parallel  with  other  kinds  of  learning,  in  the  kind  of  skill  and  knowledge 
acquisition  of  which  members  attending  the  workshop  are  interested.  I  believe  that  this  is  likely 
because  I  highly  doubt  (though  not  impossible)  that  such  a  'primitive'  process  would  be  turned 
off  when  one  engages  in  more  explicit  types  of  skill  and  knowledge  acquisition.  In  other  words, 
when  weather  forecasting,  for  example,  are  the  forecasters  relying  upon  implicit  knowledge  of 
the  spatio-temporal  correlations  of  cues  and  patterns  in  addition  to  their  explicit  knowledge  of 
physics? 
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Robert  Proctor,  Ph.D. 
Purdue  University 


As  part  of  the  study  of  acquisition  and  retention  of  skill,  it  is  important  to  understand  how 
training  and  practice  influence  fundamental  information-processing  components.  Such 
understanding  can  be  gained  from  use  of  basic  tasks  that  isolate  perceptual,  cognitive,  and 
motoric  components  of  skill.  One  advantage  of  basic  tasks  is  that  learning  occurs  quickly  and 
careful  manipulation  of  variables  allows  characteristics  of  the  skill  components  to  be  discovered. 


As  an  example,  much  of  our  research  has  focused  on  transfer  of  newly  acquired  associations  in 
tasks  that  require  a  rapid  choice  response  to  a  stimulus.  In  the  prototypical  procedure,  a  two- 
choice  reaction  task  of  pressing  left  and  right  response  keys  mapped  incompatibly  to  stimuli  in 
left  or  right  locations  (e.g.,  press  “left”  key  to  a  stimulus  that  appears  on  the  “right”)  is  first 
performed.  This  is  followed  after  a  delay  by  performance  of  another  two-choice  reaction  task  in 
which  subjects  respond  to  a  nonspatial  stimulus  attribute  (e.g.,  color;  the  Simon  task)  and 
stimulus  location  is  irrelevant.  Thus,  the  spatial  mapping  used  in  practice  is  task-irrelevant  in  the 
transfer  task.  Without  such  practice,  responses  are  faster  and  more  accurate  when  the  stimulus 
and  response  locations  correspond,  which  defines  the  Simon  effect.  However,  after  performing 
the  incompatible-mapping  task,  the  advantage  of  the  spatially  compatible  responses  in  the  Simon 
task  is  eliminated.  This  outcome  implies  that  subjects  acquired  incompatible  stimulus-response 
associations  and  transferred  them  to  a  subsequent  task  even  though  they  were  no  longer  relevant. 


Through  manipulating  several  variables,  we  have  found  that: 

1.  The  incompatible  stimulus-response  associations  are  retained  for  at  least  a  week. 

2.  The  strength  of  the  associations,  as  implied  by  the  magnitude  of  the  transfer  effect, 
increases  as  the  amount  of  practice  trials  increases. 

3.  These  associations  have  both  modality  specific  and  general  components,  as  practice 
with  auditory  stimuli  produces  some  transfer  to  a  visual  Simon  task. 

4.  With  extended  practice,  more  abstract  rule-like  procedures  are  acquired  that 
generalize,  for  example,  across  different  spatial  orientations. 

5.  Practice  with  left  and  right  pointing  arrows  transfers  to  stimuli  varying  in  left  and 
right  locations,  and  vice  versa,  implying  reliance  on  common  visuospatial  codes. 

6.  Arrows  also  show  some  indication  of  producing  transfer  to  location  words  (and  vice 
versa)  after  more  extended  practice,  whereas  spatial  locations  do  not,  suggesting  that 
arrows  also  activate  verbospatial  codes  common  to  verbal  stimuli. 

7.  No  reduction  in  transfer  occurs  when  perceptual  features  of  response  devices  used  in 
the  practice  and  transfer  sessions  are  altered,  but  significant  reduction  occurs  when 
responses  are  executed  differently  (by  pressing  keys  with  two  hands  or  moving  an 
index  finger  to  a  response  key)  in  practice  and  transfer  sessions.  These  results  suggest 
that  motoric  or  procedural  components  are  more  salient  than  perceptual  components. 
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Basic  tasks  of  this  type  can  be  used  to  investigate  and  test  characteristics  of  learning.  The  results 
can  be  used  to  develop  and  test  quantitative  and  computational  models  of  skill  acquisition, 
retention,  and  transfer  for  predicting  training  effectiveness. 
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Karen  Velkey 
Northrop-Gtumman 

Key  Issues  in  Expertise  and  Retention  of  Learning  and  Expertise: 

•  Ability  to  quickly  make  deep  and  meaningful  connections  between  previous  learning  and 
new  or  novel  situations. 

0  I  think  this  requires  “deep”  learning  of  the  original  concepts  and  repeated  and 
deliberate  practice  in  making  those  connections. 

•  Do  we  teach  people  how  to  be  mentors  and/or  give  them  specific  tools  to  share  their 
expertise  in  a  meaningful  way? 

0  We  may  not  be  giving  learners  the  best  opportunity  to  become  experts  if  we  don’t 
also  train  mentors  and  experts  in  helping  that  person  learn.  It  seems  like  many 
mentorship  programs  rely  on  a  “watch  and  learn”  mentality  where  the  deeper 
knowledge  of  the  mentor  may  not  be  obvious. 

0  I  have  found  over  the  years  that  the  most  experienced  and  knowledgeable  people 
often  do  a  terrible  job  of  describing  what  it  is  they  do.  Simply  put,  they  are 
experts  because  they  make  the  job  look  “easy”.  They  may  need  training  to  really 
be  able  to  express  the  complexities  of  their  job  that  have  become  second  nature. 

•  Do  we  reward  /  recognize  people  for  the  mentoring  aspects  of  their  work?  That  is,  do  we 
reward  them  based  on  how  well  their  mentees  perform? 

0  Many  experts  have  been  rewarded  over  long  careers  for  being  the  expert  and 
sometimes  THE  expert  in  a  particular  area.  Making  the  transfer  of  that  data  a 
rewarding  experience  both  personally  and  professionally  could  be  important  as 
well. 

•  It  seems  that  we  have  to  allow,  and  even  insist,  that  people  learn  in  a  variety  of  ways  and 
be  able  to  translate  that  learning  into  as  many  relevant  contexts  as  possible. 

•  Is  there  value  in  teaching  novices  how  to  be  mentors?  So,  not  only  teaching  them  the 
skills  in  which  we  desire  them  to  become  experts,  but  teaching  from  the  very  start  how  to 
describe  the  skills  they  are  learning  and  share  knowledge  with  others. 
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David  Ziebell 

Electric  Power  Research  Institute 


The  critical  expert  is  an  Agent  of  the  socio-technical  entity  (as  opposed  to  journeyman)  as  a 
cognizant  in  the  socio-technical  entity. 

Assume  the  relevant  socio-technical  entity  is  an  adaptive  system  within  an  ecology  of  adaptive 
systems,  assume  truth  of  the  notion  of  7  elements  of  adaptive  systems,  “Agent”  is  an  adaptive 
individual  with  power  (connection,  authority)  over  the  larger  system  (this  is  a  proposed 
definitional  notion). 

Contrast  with  mindfulness  and  decision-making  in  which  the  journeyman  cog’s  mindfulness  is 
about  the  task  and  the  environment  and  the  decisions  are  localized,  merely  homeostatic  with 
respect  to  the  larger  entity’s  purpose/mission. 

The  expert  Agent,  however,  has  awareness  about  the  larger  entity,  and  can  change  the  mission, 
re-configure  the  entity,  perhaps  even  cause  the  entity  to  navigate  the  environment.  Analogy  with 
the  default  mode  function/executive  function  of  the  brain  (self-consciousness  vs.  consciousness). 

Is  a  hiatus  from  duty  (for  an  expert  Agent)  similar  to  unplugging/disconnecting  the  default  mode 
function  in  the  brain?  If  so,  the  problem  of  re-immersing  after  a  hiatus  may  be  more  one  of  re¬ 
connection  with  an  evolved  entity  and  less  the  notion  of  rebuilding  skills. 

Towards  methods  at  this  level:  Disconnection  is  not  really  sleep,  but  perhaps  dreams  as  “serious 
games?”)  Examples  in  industry:  Special  Projects  VP,  relay  protection  asset  manager,  distribution 
system  chief  district  operator.  Seen  from  the  perspective  of  the  individual’s  lifespan/career,  the 
change  from  cog  to  agent  is  not  a  developmental  phase  so  much  as  a  transformation. 
Novice/joumeyman/master  progression  as  private  victories,  becoming  an  agent  is  a  public 
victory  (after  Covey). 

Selection  criteria  for  prospective  “agent”  may  be  different?  What  is  the  antidote  for  the  Peter 
Principle?  Does  this  systems  view  provide  a  basis  for  a  better  differentiation  between  “resilience 
and  adaptivity?”  Homeostatis  and  stability  provide  near  term  continuity,  at  the  point  of  the  spear 
“adaptivity”  is  valued  as  being  the  ability  to  react  to  environmental  insult  and  still  complete  the 
given  mission. 

But  true  creativity  (where  the  Agent  works  at  the  socio-technical  entity  level)  involves  stability 
into  the  future,  growth,  evolution  in  a  co-evolving  ecology.  This  is  more  than  homeostasis,  it  is 
life  and  the  procreation  of  the  American  way.  Requisite  imagination  in  the  war  on  terror? 

Contrast  “two  kinds  of  transfer”  with  “transfer”  vs  “transformation  of  scope.”  Need  a  better  way 
to  say  this  (the  notion  or  transformation  to  agent,  which  may  be  a  “law  of  nature”  at  a  level  of 
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complexity  that  human  beings  normally  don’t  experience  directly.)  This  transformation  may  be 
akin  to  a  peak  experience,  in  which  all  of  one’s  tools,  knowledge,  experience,  and  environment 
feel  like  extensions  of  one’s  body  -  the  pilot  is  the  airplane  and  the  airspace  and  the  bullet?  The 
chief  district  operator  is  the  procedures,  the  control  room,  the  manhole  and  the  electrons? 
Something  similar  to  this  transformation  may  be  applicable  at  the  cog  level  as  a  marker  of  the 
transition  from  person-as-asset  to  person- as -leader  or  some  such. 
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Tasking  &  Definition 


—  DSTAG  Tasking  - 


“The  Human  Systems  team  should  baseline  the  existing  programs 
(understand  current/planned  investments  including  cognitive 
techniques)  and  identify  new  ideas/technologies  to  be 


pursued.” 


What  is  “accelerated  learning"  ? 


~  The  TFT’s  Definition  ~ 


Any  learning  system  or  environment  that  attempts  to  control  for  time  spent  versus 
content  learned  with  the  following  goals; 

•  Faster  attainment  of  skill  and  knowledge  then  a  current  baseline,  and 
increase  in  on  the  job  performance  with  better  retention  of  learning 

•  Quickly  assimilate  and  convert  to  training  content  battlefield  lessons  learned 


a  Learning  Acquisition  TRADESPACE 


Learning 

Acquisitioi 


Mid- 


Irregular 

Warfare 

learning 


career 

Jearning 
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^^The  Accelerated  Learning  Challenge 


L  J  Key  AL  S&T  Challenges  that  are 
Minimally  or  Not  Funded 


*  Understanding  the  requirements  for  proficiency  and 
how  to  accelerate  its  attainment 

*  Understanding  how  to  increase  retention  of 
competence  -  “accelerated  retention” 

*  Making  practical  serious  games  for  use  in  AL 

*  Developing  technology  for  rapid  understanding  of 
learning  problem  e.g.,  rapid  cognitive  task  analysis 

*  Determining  level  of  competence  required  for  different 
stages  of  a  war  fighter's  career 

*  Calculating  the  cost-benefit  of  implementing  AL 
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Accelerated  Learning  Principles 


Routine  practice  is  not  sufficient  for 
acceleration  of  competence.  There  needs  to 
be: 

-  A  constant  “stretching”  of  the  skill,  defined  by 
increasing  challenges  (tough  or  rare  cases) 

-  High  levels  of  intrinsic  motivation  to  work  on  hard 
problems 

-  Practice  that  provides  rich,  meaningful  feedback 

-  Practice  based  on  mentoring 

-  Provisions  for  individualized/tailored  practice  due 
to  unique  learning  styles  of  each  learner 


^  J  Example  Accelerated  Learning  13 
Techniques  from  Cognitive  Sciencel^ 

*  Optimal  spacing  of  materials 

-  Spaced  is  better  than  massed  practice  (Komell  &  Bjork,  2008;  Dempster,  1989; 
Melton.  1970;  Rohrer  &  Taylor.  2006) 

Optimal  spacing;  present  enough  repetitions  so  that  material  is  mastered,  but 
overtraining  is  an  inefficient  use  of  time 

Optimal  spacing  depends  on  how  long  material  should  be  retained  for  -  optimal 
ISI  interval  =  10-20%  Rl  interval  (Rohrer  &  Pashler,  2007) 


*  Using  tests  to  improve  learning,  not  just  assessment 

Advantage  of  study-test  over  study-study  (McDaniel.  Roediger,  &  McDermott. 
2007) 

The  importance  of  active  retrieval  for  learning  (Karpicke  &  Roediger,  2008) 


•  The  benefits  of  self-explanation  (Chi,  1996,  2000) 


*  Directed  comparison  for  revealing  deep  principles  (Centner  &  Namy,  2004; 
Loewenstein,  Thompson.  &  Centner,  1999;  Rittle-Johnson  &  Star,  2007) 
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AL  methods  for  IW/COIN/SSTRO  Readiness 
Training  and  for  Mid-Career  Competence  Learning^ 


Sample  Training  Guidelines  for  Developing  Proficiency 


Competent  performers  know  a  lot. 

Their  knowledge  it  highly  contextual. 

Tralnmg  mutt  provide  ncreasmgly  detaeed 
knowledge,  procedures,  prmcipies.  m  context,  with 
progressive  refinement  as  expertise  develops 

Competent  performers'  knowledge  Is 
structured. 

Provide  tunable  knowledge  tiructures  early  in 
traeeng 

Competent  performers  knowledge  1  skill  Is 
compiled  and  proceduralized 

Provide  sufficient  practice  for  expenence  to  be 
compiled. 

Competent  performers  can  work  forward 
from  underlying  phnclples  or  backwards 
from  the  end  goal.  It  depends  upon  the 
case. 

Provide  undertysig  prtnaples  as  part  of  the 
knowledge  structures  Provide  unstructured  end- 
goal  exerases  only  after  pnnciples  have  been 
learned. 

Competent  performers  can  examine  a 
broad  range  of  alternatives  or  explore  a 
tingle  alternative  deeply. 

Pracuce  environment  must  provide  for  many 
aitematives  and  must  model  them  correctfy. 

Competence  keeps  developing  even  after 
many  years  and  thousands  of 
opportunities  for  deliberate  practice. 

Provide  (oumeymarvexpert  practice  environments 
through  senulation  and  carefUty  designed 
exercises. 

First  Generation  in  AL 

-  an  Example  of  Accelerating  Competence  - 


Adaptive  Thinking:  A  key  battlefield  thinking  skill— making  adjustments  in  an 
unfolding  plan  under  dynamic  conditions  of  military  operations 


Activities  Specifically  Designed  to 
Improve  Performance 

•  Identify  desired  elements  for  expert  form 

•  Learner  performs  while  attending  to  element 

•  Coach  notes  disaepancies  from  expert  form 

•  Behavior  is  repeated  until  habitual 

•  Performance  without  attending  to  expert 
form 


Expert  Cognitive  Behaviors 

•  Keep  a  focus  on  the  mission  and  higher  intent 

•  Model  a  thinking  enemy 

•  Consider  effects  of  terrain 

•  Use  all  assets  available 

•  Consider  timing 

•  See  the  big  picture 

•  Visualize  the  battlefield 

•  Consider  contingencies  and  remain  flexible 
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Captains  in  Command 


1.  Siudtntt  r*c«tv« 
training  on  the 
expert  Themes  ol 


Battlefield 
Thinking  and 
observe  model 
behavior. 


2.  Students  view  a 
3-5  minute  theme- 
based  vignette  that 
presents  a 
complex  and 
rapidly  changing 
tactical  situations. 


fgZEKSILjssrz 


3.  Students  are 
asked  to  apply 
their  tactical 
knowledge  to  think 
adaptively  and  list 
their  key 
considerations. 


4.  Students  view 
a  3D  coach  or  live 
Instructor 
discussing  expert 
considerations 
pertaining  to 
each  of  the 
battlefield 
thinking  skills. 

5.  The  students 
are  prompted  to 
evaluate  their 
response  based 
on  expert 
considerations 
and  respond  to 
coach's 
questions. 

6.  After  coverli>g 
each  of  the  eight 
themes,  students 
are  provided 
feedback  to  his  or 
her  responses 


I 


1- 


Measurement  of  Adaptive  Thinking 


Percent  of  Critical  Inrormation  Idenltfled  (by  Rank) 


u 
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Think  Like  A  Commander  &  Captains  in 
Command  Training 

Recent  evidence  indicates  that  TLAC  and  the  Captains  in  Command  training  develop  key 
battlefield  thinking  skills  comparable  to  those  exhibited  by  CPIs  with  OIF/OEF  experience 


Mean  Parcant  of 
Critical  Information  Idantifiad 


VIgnatta 


Command  Thr*  Uk*  a  Coranwidtr 

**  *  toalna  (Oactoytfl *  *  k^^HocvQaotoyd) 


-  an  Example  of  Accelerating  Competence  - 
Adaptive  Thinking  Training  “Think  Like  a  Commander' 


Thinking  that  supports: 

making  adjustments  in  an  unfolding  plan  under  the 
dynamic  conditions  of  military  operations. 


Specific  Training  Task 

The  ability  to  rapidly  and  accurately  ‘size  up*  tactical 
situations:  identify  important  considerations,  risks,  key 
information,  alternative  actions,  potential  higher-order 
effects,  and  other  factors  that  may  be  significant  in 
decision  making  during  execution. 


J.  Lussier,  2008 
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Goals  of  the  Workshop 


The  objective  of  this  effort  is  to  create  a  design  and  roadmap  for 
investigations  aimed  at  developing  robust  and  broadly-applicable 
methods  for; 

(1) Accelerating  the  achievement  of  proficiency  in  USAF  mission- 
critical  specializations  (and  analogous  domains  in  the  private 
sector),  and 

(2)  Facilitating  the  retention  of  knowledge  and  skill,  especially  for 
military  personnel  who  must  take  long  hiatuses  from  their 
domain  of  primary  expertise  in  order  to  fulfill  other  military 

requirements. 


8:45  AM 

Welcoming 

ImroduciKins  and  "Lightning  Round" 

Each  pariicipanK  has  three  minutes  to  conve>'  then  locus  area  and  their  key  ideas  of  hot 
buttons  m  issues  of  traininc,  transfer  and  retention. 

9:45  AM 

Prescnialion  1 

Dr.  Dee  Andrews 

Identifying  Interestmg  and  Viable  Research  Challenge  Ouestions  About  Acceleration  ai>d 
Retention 

•  Military  Environment 

•  Long-Term  Retention  Study 

•  Program  Roadmapping 

•  DoD  Accelerated  Learoins  Studv 

10:15 

Full  GrouD  Discussion 

I0:M)AM 

Break 

10:45  AM 

Presentation  2 

Paul  Feltovich  and  Robert  llotfman 

Acceleration  and  Retention:  Research  and  Theory 

11:50  AM 

Presentation  J 

Acceleration  and  Retention  Issues 

Lt  Col  (iina  flilger.  USAF 

Intelliccncc-Surveillancc.  Reconnaissance  Rcauirements 

12:15 

Lunch 
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1:15  PM 

Prcsentalion  4 

Acccicraiion  and  Retention  Issues 

David  Ziebcll  (BPRIi  Inl'rastmcture  and  Power  Distribution  Domains 

’;()0  PM 

Full  Group  Discussion 

2:15  PM 

Presentation  5 

Acceleration  and  Retention  Issues 

Bomber  Operations 

Col  Daniel  Walker .  AFRL 

3:00  PM 

Break 

3:15  PM 

Three  Break-(7ut  Groups 
*  Selection  of  Domains 

4:45  PM 

Break 

5:00  PM 

Full  Group  Discussion: 

*  Research  Desijin  and  Methixlology  Considerations 

•  Closing  comments  and  Liwk-ahead  at  Agenda  for  Day  2 

7:00  PM 

Group  Dinner  at  Tia  Rosa’s 

Breakfast 

9:15  AM 

(ROframing  based  on  Day  1  results 

9:30  AM 

Accelcraiiofl  and  Rctcniion  Issues 

Pilot  Transition  Training 

Ll  Col  Doy  Ic  Turner.  USAF 

10:00  AM 

Break 

10: 1 5  AM 

Three  Break-Out  Groups 

*  Selection  of  Domains 

*  Research  Design  and  Methodology  Considerations 

IhlSAM 

Break 

II  30  AM 

Acceleration  and  Rctcniion  Issues 

F-16  Rctcniion  Research 

Dr  Winston  Bcnneit 
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12:15 

Lunch 

1:15  PM 

Acceleration  and  Retention  Issues 

Dr.  Kevin  Gluck  (with  Dr.  Tiffany  Jaztrzembski) 

Predictive  Performance  Optimizer 

1 :45  PM 

Three  Break-Out  Groups 
•  Challenge  Questions  for  Research 

2:30  PM 

Break 

2:45  PM 

Full  Group  Brainstorm 

•  From  Challenge  Questions  to  a  Research  Roadmap 

3:45  PM 

Break 

4:00  PM 

Full  Group  Brainstorm 

•  From  Challenge  Questions  to  a  Research  Roadmap 

5:00PM 

Closing  Discussion:  Look  ahead 

Acknowledgements 

_ 
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Distributed  Mission  Operations 
Training  Research:  skill  Retention 

and  Decay  Study  1 

Results 

9-10  October  2009 


Winston  “Wink”  Bennett,  PhD 
711**’  Human  Performance  Wing 
Human  Effectiveness  Directorate 
Air  Force  Research  Laboratory 


Fighter  Combat  Training  and 
Management 

•  Expertise  in  this  environment  depends . 

•  No  formal  metric  for  expertise 

-  Flying  hours  typical  metric  -  500  hours  in  fighters  =  combat 
mission  ready 

-  School/training  pedigree 

-  Weapons  school  is  considered  “Graduate  School’’ 

•  Development  paths  to  “Full-up  Round”  varied,  formalized 

•  Mix  of  professional  and  ad-hoc  training  and  experience 

•  Sophisticated  career  path  and  management 

•  Tracking  in  and  out  of  flying  with  formal  reconstitution 
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The  Problem"^^ 


•  Lost  apprenticeship  -  fewer  opportunities  to  develop  expertise 

•  Training  requirements  outpacing  capacity/opportunities  to  train 

•  Training  to  gain/maintain  readiness  is  a  growing  concern 

-  Aging  systems  and  hi  ops  tempo  limiting  live  training  opportunities 

-  Range  limitations  reducing  fidelity  of  live  training 

-  High  costs  and  complexity  of  missions  increasing  costs 

•  Cross-domain  integrated  training  (air,  space,  and  cyber)  not  realistic 

•  Formal  education  and  training  system  trapped  in  1985  (on  a  good  day!) 

•  Challenge:  Are  there  mixes  of  approaches  and  technologies  for 
training  that  impact  decay  and  retention  of  skill  and  expertise? 

1:  HQ  Air  Foret  124091-  Un.  Wmu.  Can*(rwc0rtCoinctpf  oFOptnffont  <Oran.  Jwm.  20091.  W*tMr»Qten  DC.:  2:  Otnet  ollht  UnMr  stertevy  or  Otttnet  12006).  Jotni  ittmiHitmil  tcMnct  tnOtocAnoFogypran. 
VWitMTHiton.  OC.:  3:  Otptiij— ni  or  Otfontt  RtMtrch  and  Enginatrkta  12007).  Stranflttt  pttA  WulwgtOfi.  DC. 


Framing  Our  Research: 
What  We’re  Trying  to  Do 


Improve  airman  performance  and 
readiness  through  development  and 
application  of  human  learning,  instruction, 
and  performance  assessment  science  and 
technology.  Key  characteristics  .... 

-  Adaptive  to  learner  needs 

-  Instructionally  and  operationally  rich 

-  Routinely  accessible  to  warfighters 

-Seamlessly  integrated  with  live  systems 
for  continuous  learning 

-Generalizable  across  missions,  common 
approaches,  methods,  and  data 

-Sustainable  with  warehoused  knowledge 
and  managed  events 
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Examining  Retention,  Transfer, 
and  Decay  in  the  Wild 

•  Wicked  hard  stuff  to  do  in  this  content  domain 

•  Lots  of  things  competing  with  research  agenda 

•  Threats  to  validity  are  pervasive 

•  Community  interest  is  strong  but  fractured 

•  Interesting  learning  and  expertise  dynamics 

•  Difficult  to  determine  what  constitutes  expertise  when 

•  What  does  acceleration  mean  in  this  context 

•  When  do  you  have  it,  how  do  you  get  it,  how  do  you  keep 
it,  how  do  you  lose  it 


Is ‘DMiy*  REALLY 

If  you  can  racovsr  It  quiddy  >  wns  it  wsr  (raaMy) ‘‘gom’*? 


Research  and  Development  Innovation: 
Skill  Retention  and  Decay  Study 


•  What  S&T  gaps  are  we  addressing? 

—  Limited  applied  data  on  air  combat  skill  decay 

—  Quantifying  factors  that  may  moderate  decay 

—  Identifying  what  remains  stable/decays  overtime 

•  What  we’ve  done: 


—  Longitudinal,  experimental  study  with  operational  pilots 

—  Manipulated  retention  interval  (3  or  6  months) 

—  Examined  initial  training  gains,  skill  decay,  skill  reacquisition 
rate,  and  moderators  of  skill  decay 

•  What  impact  can  the  S&T  have? 

—  Inform  frequency  of  DMO  training  requirements 

—  Identify  what  skills  decay  faster  than  others  -  shelf  life 

—  Quantify  potential  moderators 
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Methods  &  Measures 


•  Longitudinal  study  using  operational  F-16  pilots 

•  Initial  5-day  DMO  structured  training  visit,  followed  by  a 
2-day  return  DMO  training  visit.  Demographics  captured 
each  visit 

•  3  or  6  month  retention  interval 

•  Equally  complex,  mirror-image  point  defense  missions 
flown  at  beginning  and  end  of  first  and  second  visit 

•  Performance  assessed  using  objective  measures 

-  Mission  outcomes:  strikers  to  target,  enemy  aircraft  killed, 
friendly  mortalities,  weapon  hit  ratios,  Top  Gun,  etc. 

-  Process/skill  measures:  Time  in  vulnerability  regions  (e.g.,  MAR, 
N-pole),  weapons  employment  measures  (e.g.,  ranges,  mach,  loft, 
F-pole,  etc.),  comm  coordination  (e.g.,  voice  overs),  etc. 


Expectations 


•  Initial  week  training  gains  will  be  consistent  with  findings 
from  AFRL’s  DMO  within-simulator  training  effectiveness 
baseline  study 

-  There  should  be  significant  improvement  in  performance  during 
the  five  days  of  initial  training 

•  There  to  be  some  skill  decay  during  the  retention  interval 

-  The  amount  of  decay  should  depend  on  the  length  of  the  decay 
interval,  i.e.,  the  3-month  group  should  demonstrate  less  skill 
decay  than  the  6-month  group 

•  Both  groups  should  show  significant  improvement  in 
performance  during  the  refresher  training  -  ideally 
recovering  to  the  level  of  skill  attained  during  the  initial 
training,  or  possibly  exceeding  that  level 
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DMO  Skill  Retention  Study:  gg 

Procedure 


Teams 


•  F-16  Pilots  were  allowed  practice  runs  with  the  simulator  for 
an  hour. 


•  Pilots  were  then  “benchmarked”  or  tested  on  their  air-to-air 
point  defense  scenario  performance  on  Monday,  then  tested 
again  on  Friday.  Five-day  initial  syllabus 

•  Pilots  were  then  tested  twice  on  their  2-day  return  trip  either 
three  or  six  months  later 


Anticipated  Skill  Retention  Results 


Training  gain 

After  3  month  decay  interval 
““After  6  month  decay  interval 
- Expected  end  performance 


10 
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Anatomy  of  Example  Scenarios 


MEC  Definition/Validation 

-  MEC  Analysis  (developmental  experiences  and  training  emphasis  areas  Illuminated) 

-  Event  design  Is  framed  by  these  developmental  experiences 

-  Map  experiences  back  to  Knowledge  and  Skills  (K/S’s) 

-  Fly  out  and  ensure  trigger  events  tap  K/S's 

■  Incorporate  Into  syllabus  with  elaboration  and  deliberate  practice  approaches 
Event  1  Event  2 


Experiences  -  Daytlme/supersonlc  employment, 
operations  against  threat  w/chaff/flare,  radar 
search,  targeting  responsibilities,  1:1  force  ratio 

.  Skills  -  Triggered  by 
time/range  and 

^  »  include  adapts  to 

/  threat  changes, 

/seam  010"  at  intarpr.ts  sensor 

/  30NM  lor  60  aec  output., 

/  and  descend  121^ 


26K 


25K 


Knowledge  -  Commit  criteria,  comm  standards, 
engage  criteria,  formation.  ROE.  threat  capabilities 


Experiences  -  Previous  scenario  plus  a  full  range 
of  adversary  threats,  task  saturation,  operating 
area  restrictions.  1:3+  force  ratio 

Skills  -  Trioeers  here 
“build  on”  previous 
adding  builds  picture, 
listens,  selects  tactic, 
manages  stress, 
radar  mechanics 


Knowledge  -  Previous  scenario  plus  follow>on 
options,  friendly  capabilities,  package 
composition,  mission  phases 


Demographics 


Participant  dcmoj’raphics,  3-mtinth  group 


Rank 

N 

Avg.  age 

Avg.  years  of  serv  iee 

Avg.  F-16  Hours 

First  I.iculcnam 

5 

26.2 

4.5 

241.3 

Captain 

14 

31.4 

8.4 

819.0 

Major 

5 

36.0 

15.2 

1201.6 

No  rank 
identified 

4 

1340.5 

Total 

28 

31.3 

9.0 

884.2 

Participant  demographics,  6-month  group 


Rank 

N 

Avg.  age 

.\vg.  years  of  serv  ice 

.•\vg.  F-16  Hours 

First  Lieutenant 

4 

27.5 

5.0 

237.3 

Captain 

21 

29.5 

8.2 

668.3 

•Major 

10 

36.1 

14.5 

1307.6 

Lieutenant 

Colonel 

6 

38.8 

18.5 

1876.9 

Total 

41 

32.3 

10.9 

959.1 

12 
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SOME  SELECTED  OUTCOME 
MEASURES  FOR  CONSIDERATION 
. JUST  A  TASTE 


Study  Completed,  Other  Analyses  Are  In 

Progress 


t3 


Enemy  Strikers  Reaching  Target 

Data  From1 8  teams  (72  pilots) 


This  is  an  outcome  measure:  How  many  bombers  are  killed  before 
bombing  a  friendly  target 

!  Performance  improved  during  initiai  training,  decayed  somewhat  over  the  retention 
interval,  and  remained  reiativeiy  constant  during  refresher  training.  Only  the  initial 
training  gain  was  statistically  significant  in  the  MANOVA  analyses 
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Top  Gun  Score 

Data  From18  teams  (72  pilots) 


C 


6 

o 


c 

O) 

E 

Q) 

> 

2 

a. 


E 


c 

Q} 

U 

w 

a> 

Q. 


A  summary  metric  for  performance  on  a  single  point  defense  scenario 

•  Enemy  strikers  reaching  target  (-300  points) 

•  Killing  enemy  bombers  before  target  (+450) 

•  Killing  enemy  bombers  after  target  (+150) 

•  Enemy  fighters  killed  (+150) 

•  Fratricides  (-900) 

•  Any  other  F-16  mortalities  (-300) 


-50 


Initial  training 
Pre  Post 


Refresher  training 
Day  1  Day  2 


This  is  outcome  measure:  A  composite  score  for  how  well  they  did  based  on  a 
predetermined  set  of  scoring  criteria  (Range  is  -1800  to  +1800  points) 

!  Performance  improved  during  initial  training,  decayed  during  the  retention  interval,  and 
unexpectedly  continued  to  decay  during  refresher  training.  Only  the  initial  training 
gain  was  statistically  significant  in  the  MANOVA  analyses  ts 


Mach  at  Pickle 

Data  FromIS  teams  (72  pilots) 


This  is  a  process  measure:  The  speed  of  the  jet  at  time  of  missile  launch 

!  There  is  little  absolute  change  in  this  metric  during  initial  training,  retention  interval  or 
refresher  training.  MANOVA  analyses  did  not  reveal  any  significant  changes.  However,  the 
differences  observed  are  worth  more  explanation  due  to  the  importance  in  actual  combat 
(i.e.,  practical  significance)  le 
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Some  Initial  Conclusions 


•  First  week  of  study  replicated  findings  from  previous 
within  Simulator  Study  -  Large  DMO  training  week  effects 

•  Some  evidence  of  Skill  Decay  across  retention  period 

-  Likely  retention  period  impacts  on  decay 

-  Pilot  level  of  experience  moderated  skill  decay  (Hi  experience) 

•  Results  statistically  weaker  than  desired 

-  Small  sample  size  provided  very  weak  statistical  power 

-  Team  performance  on  an  extremely  complex  and  dynamic  task 

-  Some  circumstantial  evidence  that  teams  may  have  known  the 
benchmarks  they  were  going  to  fly 

-  Other  mitigating  variables  not  tracked 

-  Flight  lead  experience  may  be  mitigating  overall  team  decay 

•  Analyses  show  decay  at  the  team  level,  analyses  need  to 
be  conducted  to  investigate  individual  level  decay  (e.g. 
crank  mechanics) 


Design  Limitations 

•  Current  study  designed  as  a  very  conservative  test  for 
skill  decay 

(Seemed  like  a  good  idea  at  the  time!) 

•  Factors  making  study  conservative: 

-  Highly  experienced  pilots 

-  Most  experienced  pilot  flew  as  lead 

-  Used  performance  for  stable  teams  of  pilots,  less  likely  to  have 
team  cohesion  3  or  6  months  later  in  reality 

-  Pilots  receive  continuous  training,  so  retention  intervals  had 
variable  additional  training 

-  Good  argument  that  no  change  (null  effect  default)  is  possibly 
not  the  best  statistical  test.  Some  degree  of  improvement 
should  be  used 

-  Used  benchmark  tasks  that  were  performed  near  ceiling  at  end 
of  training  week,  i.e.  assessments  did  not  map  full  range  of 

skill  levels  i> 
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Lessons  Learned  For  Future 


•  Need  larger  samples  for  this  kind  of  study 

•  Greater  experience  range  with  content  domain 

•  Additional  retention  intervals 

•  Must  track  what  happens  in  the  retention  interval  - 

clearly  practice  and  refreshing  was  ongoing  -  we 
attempted  to  track  this  experience,  but . 

•  Develop  several  sets  of  pre-  and  post-test  scenarios 
that  are  not  part  of  regular  research  weeks 

•  Examine  other  metrics  that  might  be  more  sensitive 
(e.g.,  shared  understanding,  tactical  knowledge) 

•  Variety  of  outcome  and  process  measures  in  our  data 
set  still  being  examined 


Some  Closing  Thoughts 


•  Research  of  this  kind  is  crazy  hard,  but  very  important 

•  Two  combat  theaters  compete  for  subjects  and  impact 
schedule 

•  Examining  training  alternatives  and  their  immediate  and 
longer  term  impacts  of  expertise  and  retention  is  critical 

•  Accelerating  competence  (expertise)  only  works  if  the 
environment  supports  and  promotes  growth 

•  Accelerated  learning  interventions  must  be  systemic 

-  Point  solutions  will  not  be  effective  in  the  absence  of 
opportunities  to  practice  and  extend  knowledge/understanding 

-  Monitoring/tracking  mechanisms  must  be  part  of  solution 

!  Effective  training  may  not  be  efficient,  and  vice  versa  in 
terms  of  developing/sustaining/growing  expertise 
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Thanks  a  BuncKi 
Questions^ 

Briefer  Name;  Dr  Winston  “Wink”  Benne 
Phone  (com):480. 988. 6561,  extensloh^SW^pi 
Emaihwinston. bennett@mesa.afmc.af.mil 
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Predictive  Performance  Optimizer 

10  October  2009 


Kevin  Gluck,  PhD 
Senior  Research  Psychologist 
Human  Effectiveness  Directorate 
71 Human  Performance  Wing 
Air  Force  Research  Laboratory 
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Outline 


I.  Historical  Background 

II.  Approach  and  Validation 

III.  Predictive  Performance  Optimizer 

IV.  Future  Directions 
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«»» 

•  Stat  Lady 

•  SMART  Algorithm 

(Student  Modeling  Approach  for  Responsive  Tutoring) 


1994 


Shute,  V.  J.  (1995).  SMART:  Student  Modeling  Approach 
for  Responsive  Tutoring.  User  Modeling  and  User- 
Adapted  Interaction,  5,  1-44. 
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1997 


•  Decomposition  of  learning  curve  (Masters  thesis) 


Scenario 


w 


1999 


Dissertation  -  eye  movements  and  algebra  tutoring 
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January  2000 
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Cognitive  Tutors™ 


C'  S^CDtiAtryMBfli  CurnruMiiilIh  Proven  SuriusK;  Cognirivc  lulor.'^5cirtvurc(Xcnm/  MlrrotQmnrerncMwfii^'i  r  pron 


-■Aji. J>.\.. Jiv  <Ji  1 


;*♦  X  ;  J 

• 


■  Carnegie  Learning 


RESCMC-ftAiE  MATHCUkiaCUU  PROFEtSOnML  DCVaOHtCm  NVOWSTArE  UjCCBSrOUES  AROITUS 
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Can  We  Use  the  Cognitive  Tutors  ® 
as  Cognitive  “Trainers”? 


•  Successfully  used  in  algebra  educational 

•  Cognitive  skill  represented  as  production  rules 

•  Mastery  estimate  is  adaptation  of  Bayes’  theorem 

•  Assumes  a  2-state  learning  model. 

•  Probability  that  a  rule  is  in  the  learned  state  following  n^^ 
opportunity  to  apply  that  rule: 

piLJ  =/j(i:„_i|eAldence)  +  (1  -/j(X„_i|eAidence))  *  piT) 


/)(leai'ned)  =  />(learned  last  time)  +  /j(not  learned  last  time)  */»(ti'ansition) 
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Limitations  of 

Cognitive  Tutor™  Approach 


•  Does  not  account  for  skill/knowledge  decay 

-Once  learned,  it  is  assumed  learned  forever 

-OK  (perhaps)  for  tracking  short-term  learning  in 
educational  domains 

-  Not  good  for  predicting  long-term  performance 
profiles  in  military  training  domains 

•  Does  not  predict  response  times 

-Not  mapped  to  retrieval  latencies  or  cognitive  cycle 
time 

-OK  for  static  educational  domains 

-  Not  good  for  time-critical  m  ilitary  domains 


331 


2004  -  Proposed  Alternative 


•  A  general  equation  for  learning  and  forgetting  (Anderson 
&  Schunn,  2000): 

Performance  =  A  * 

A=  Scaling  parameter 
N  =  Amount  of  practice 
c  =  Learning  rate 
T  =  Time  since  learning  occurred 
d  =  Decay  parameter 

•  Equation  provides  reasonable  fits  across  different  delays 
and  tasks  (see  examples) 


11 


Memory  for  Street  Names 


Bahrick  et  al.  (1 983):  T able  8 


12 
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Relearning  Nonsense  Syllables 


Ebbinghaus  (1885/1964):  Chapter  7 


Relearning  Sequential 
Control  Operations 


Ammons  et  al  (1958):  Figure  3 
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Challenges  to  this  Approach 


•Simple  performance  equation  does  not 
account  for  distribution  of  training  over  time 

-All  learning  is  assumed  to  occur  at  T=0 


•However,  training  is  always  distributed  to 
some  extent,  and  many  times  distributed 
across  days,  weeks,  or  years 

-Spacing  effect 


15 


Outline 

_ 

I.  Historical  Background 

II.  Approach  and  Validation 

III.  Predictive  Performance  Optimizer 

IV.  Future  Directions 
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2005  -  Began  work  on 
incorporating  the  spacing  effect 


•  Leverage  over  100  years  of  research  in  experimental, 
mathematical,  and  cognitive  psychology 


•  Showcase  new  innovations  (a  priori  predictions  and 
principaiiy-guided prescriptions)  OMQV  existing  State-Of-the- 

art  ('Anderson  &  Schunn,  2000;  Pavlik  &  Anderson,  2006;  Lindsey,  et  al.,  ICCM  2009;  van  Rijn, 
etal.,  ICCM2009j 


■  Validate  model  implementations  and  analysis  tools 
using  human  performance  data  from  as  broad  a  range 
of  contexts  as  possible 


17 


Predictive  Performance  Equation 


S  =  scaling  parameter 


aised  to  -d,  the  decay  ratg^ 

Performance  =  T 

T  =  time  since  learning 


lag  /> 


Stabiiity  Term: 
Experience  amassed  as  a 
function  of  training  distribution 
^and  time  passed  -  captures 
spacing  effect 
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The  Spacing  Effect 


Scenario  1: 


Scenario  2: 


Hi5tor«ai  and  Prediclivc  HcrTormanoc  | 

§ 

.  u 

Poorly-Spaced,  Massed  Training  Regimen 


Histc«tcd  ana  Pradlctiv*  Psftorrnanca 


Conclusion:  Principled  spacing  of  baseline  training  produces 
durable  retention  and  reduces  the  need  to  retrain  or  reacquire. 


Validation  with  Data  from 
Psychological  Literature 


120 
S  100 

o  80 
Hi 

“  60 
8  40 
£  20 
0 


Hnoviledge  Retention 

Bdftrick  6t^/.,1993 


y 


-  Humans 
•  Model 


63  203  343  365  730  1095  1825 

Days  Sirce  Learring 


336 


V.  ^Objective  Performance  Measurement 
in  the  DM0  Testbed 


Validation  with  Data 
from  the  DMO  Testbed 


•Validation  effort  underway  in  highly  applied/relevant  domain 

-  Pilot  teams  composed  as  4-ships 

-  Complete  a  week  of  simulator  training  research  and  return 
either  3  or  6  months  later  to  be  reassessed 
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Validation  with  Data 
from  the  DMO  Testbed 


-Pilot  performance  is  assessed 
during  a  training  week 


-Identification  of  meaningful  learning  variables  is  key! 


Validation  with  Data 
from  the  DMO  Testbed 


-Mode!  tracks  knowledge  and  skills  of  learner  during 
initial  training  -  parameters  uniquely  calibrated  at  this  time 


24 
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Validation  with  Data 
from  the  DMO  Testbed 


-Extrapolates  mathematical  regularities  of 
knowledge  and  skill  changes  to  use  in  a  predictive  fashion 
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Validation  with  Data 
from  the  DMO  Testbed 


-Model  accurately  predicted  pilot 
team  performance  to  a  high  degree  of  precision 


26 
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Outline 


I.  Historical  Background 

II.  Approach  and  Validation 

III.  Predictive  Performance  Optimizer 

IV.  Future  Directions 
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On  the  Nature  of  Applications 

-  Allen  Newell  (1990) 


“Applications  are  an  important 
part  of  the  frontier  of  any 
theory...” 
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Predictive  Performance  Optimizer 
Controls  &  Demonstration 


iTrammgGoalt  Regimen  Definition* 

Generel  Treraig  Go*l: 

No  oplnuer  goel  selected  w 

Regmen  start  date: 

02/14/2009  11:00  4 

Performance  Objectve  level:  6.158  4- 

Regmen  total  length: 

Performence  Sustaned level:  12.316  4- 

350  :  days 

Active  regmen  values 

Traraig  Budget:  «  4- 

Total  events:  9 

Performance  Goal  Date:  09/23/2009  22:‘M  4- 

Performance:  24.77 

'Bockl|Bk>ck2|ekidi3|aod[4|&k)Cfcs|eiod(6|Bk)ck7 

Blodt8|aodt9 

Traevig  Uodc  length; 

1 : 

days 

Hsson  diration; 

l4-  hoirs 

Lag  fetotMng  blodc 

^3  4- 

days 

Gap  between  msMns; 

34  hours 

Hssons  per  day: 

1 : 

1st  mssem  start  tsne: 

11:004- 

Add  New  Bock 
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Model  Sensitivity  to  the 
Unique  Needs  of  the  Learner 
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Model  Sensitivity  to  the 
Unique  Needs  of  the  Learner 


Parameter 

Team  115 

Team  112 

Stability  Term 

]  46 

1.46 

Scalar 

0.064 

0.164 

Learning  Rate 

0.999 

0.729 

Decay  Rate 

0.068 

0.001 
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V  J  Prescribe  Scheduling  of  Refresher 
Training  to  Maintain  Readiness 


Well-Spaced,  Distributed  Baseline  Training 
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K^J  Prescribe  Scheduling  of  Refresher 
Training  to  Maintain  Readiness 


Poorly-Spaced,  Massed  Baseline  Training 


33 


Exploring  Predicted  Performance 
Implications- 


Predictive  capability  can  help  determine  which  future 
training  regimen  would  best  achieve  training  goals 


Histoiical  and  Predictive  Performance 
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Outline 


I.  Historical  Background 

II.  Approach  and  Validation 

III.  Predictive  Performance  Optimizer 

IV.  Future  Directions 
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Looking  Ahead... 


•  Validation  in  the  Field:  AETC  and  Live-Fly  Training 
Contexts 


•  Confidence  intervals  and  Mission  Risk 


•  Neurobiology  of  learning  and  forgetting 
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Thank  You! 


Dr.  Kevin  Gluck 
Kevin.gluck@mesa.armc.af.mil 
(480)  988-6561  x677 
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UNCLASSIFIED 


Acceleration  &  Retention 
Issues:  ISR  Example 


Lt  Col  Gina  Hilger 
Chief,  ISR  Requirements  &  Ops  Div 
US  Air  Forces  Central,  Shaw  AFB 


Overall  Classification  of  this  Briefing:  UNCLASSIFIED 

UNCLASSIFIED  1 


Overview 


■  Career  Path  Broadening 


■  Reintegration  Challenges 


■  Thoughts  on  Ways  Ahead 


UNCLASSIFIED 


2 
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UNCLASSIFIED 


Career  Path  Broadening 

External  &  Internal 


m  Career  broadening  encouraged  in  Air  Force  (external) 

■  Broader  exposure  to  different  careers/functions/levels  designed  to 
develop  better  senior  Idrs  capable  of  integrating  complex  missions 

■  Often  a  stratifier  for  promotion 

■  Professional  military  education  &  higher  degrees  a  must 

■  Many  positions  are  not  affiliated  w/  a  career  field,  but  must  be  filled 
(air  attache,  PME  instructor,  legislative  liason,  ROTC/CC) 

■  New  intelligence  career  field  guidance  (internal) 

■  3  X  areas  of  core  expertise  for  all  Intel  officers  (analysis,  global  ISR 
operations,  effects-based  ISR  planning) 

■  6  X  professional  competencies  (analysis,  foreign  area  expertise, 
global  ISR  enterprise,  targeting,  AOC  ops,  programming/acquisition) 

■  Consecutive  assignments  in  same  competency  by  exception  only 

■  Should  have  at  least  two  different  jobs  during  one  assignment  as  well 

■  Broadening  within  the  Intel  career  field  is  intentional 

UNCLASSIFIED  3 


UNCLASSIFIED 


Reintegration  Chaiienges 
(1  of  2) 


■  Difficult  to  stay  in  tune  w/  primary  career  field  during  broadening 

■  Broadening  Jobs  are  just  as  time  consuming  as  career  field  positions 

■  Current  supervisors  demand  your  full  attention  on  task  at  hand 

■  Facilities/systems  access  often  not  available  to  keep  in  touch  (eg, 
classified  material/buildings/badges,  need-to-know  clearance) 

■  Reintegration  as  a  CGO  may  not  be  as  difficult  if  return  to  same 
competency  (not  guaranteed) 

■  Reintegration  as  an  FGO  much  more  challenging 

■  Greater  supervisory/leadership  responsibilities 

■  Limited  time  to  focus  on  learning  /  reintegrating  (high  ops  tempo) 

■  Slave  to  the  "task  master;”  always  putting  out  fires 

■  Multi-tasking  is  the  norm;  no  large  chunks  of  time  to  study  at  work 

■  Burden  of  reintegrating  often  on  the  individual;  no  deliberate  program 

■  Fewer  FGOs  in  unit  -  can’t  afford  to  lose  one  for  lengthy  trng  courses 

■  Tendency  to  seek  &  to  be  given  tasks  that  fall  within  prior  expertise 


UNCLASSIFIED  4 
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UNCLASSIFIED 


Reintegration  Chaiienges 
(2  of  2) 


■  “No  one”  knows  you  anymore  when  you  return  to  field 

■  Previous  mentors  no  longer  available  (many  retired);  need  new  ones 

■  Previous  social  network  at  AO-level  no  longer  intact 

■  Must  often  re-build  strong  reputation  again  within  career  field 

■  Managing  subordinate  &  other  expectations 

■  Harder  to  ask  for  help  (particularly  when  return  as  an  FGO) 

■  Subordinates  expect  you  to  be  an  “expert”  based  on  time  in  service 

■  Joint  and  component  counterparts  also  expect  you  to  be  an  "expert” 

■  Sense  of  frustration  &  initial  loss  of  confidence 

■  Competitively  selected  for  career  broadening  opportunities,  but  may 
negatively  impact  career  progression  within  own  field 

■  Some  thoughts  of  “getting  out”  &  starting  2"'*  career  early 

■  Programmatic  challenges 

■  Too  few  slots  at  training  courses;  often  must  be  deploying  to  attend 


UNCLASSIFIED  5 


UNCLASSIFIED 


Thoughts  on  Ways  Ahead 
(1  of  2) 


■  “Just  in  time”  training  courses  in  intel  do  exist 

■  IFTUs  (space,  AOC,  platform,  DCGS,  AFSOC,  targeting) 

■  Need  to  increase  number  of  slots 

■  Reintegration  must  be  more  deliberate,  at  least  in  intel  field 

■  Need  mechanism  to  flag  those  officers  returning  to  the  field 

■  Add  training  line  remarks  to  billets  for  returnees 

■  Schedule  course  attendance  enroute  to  next  job  to  ensure  it  happens 

■  Treat  returnees  similar  to  those  shifting  intel  core  competencies 

■  Developmental  Team  (DT)  vectors  can  address  expertise  acquisition 

■  Intelligence  Master  Skills  Course  (IMSC) 

■  Good  for  “breadth;”  consider  sending  FGO  returnees  back  to  IMSC 

■  Computer  Based  Training  (CBTs) 

■  Good  for  exposure,  but  not  deep  comprehension 

■  Time  available  a  factor 


UNCLASSIFIED  6 
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UNCLASSIFIED 


Thoughts  on  Ways  Ahead 
(2  of  2) 


■  Deployments  build  expertise  quickly;  learning  accelerated 

■  Fewer  distractions;  long  periods  of  time  to  focus 

■  Hands-on  appiication  of  knowiedge  aids  expertise  retention 

■  Need  high  fidelity  training  courses  that  replicate  oper  environment 

■  Reintegration  must  occur  primarily  during  duty  hours 

■  Off-duty  time  needed  to  recover  from  high  deployment  rate  (1:1  dwell) 

■  Retention  factor  -  “burn  out”  on  the  horizon  for  many 

■  Tailor  training  programs  to  individual’s  preferred  learning  style 

■  Voluntary  “Warrant  Officer”program 

■  Observation;  Majority  of  Intel  Subject  Matter  Experts  (SMEs)  are 
civilians/  contractors  (more  time  to  study  intelligence  literature  & 
develop  intuition) 

■  Two  tracks:  One  for  leadership/command,  one  for  subject  expertise 

■  Need  to  study  what  level  of  expertise  is  required  to  lead  a  team  of 
experts  vice  being  the  expert  yourself  (managerial  expertise) 

UNCLASSIFIED  7 


UNCLASSIFIED 


Questions? 


Overall  Classification  of  this  Briefing:  UNCLASSIFIED 


UNCLASSIFIED 
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Accelerated  Learning  and  Facilitated  Retention 


Paul  J.  Feltovich 
Robert  R.  Hoffman 
Institute  for  Human  &Machine  Cognition 

Steve  Fiore 

University  of  Central  Florida 
Gary  Klein 

Klein  Associates  Division  of  ARA 


ihmc 


Outline 


1 .  Issues 

2.  Empirical  Base  (academe  and  “real  world”) 

3.  Theories  &  Hypotheses 

4.  Phenomena  of  Training  &  Transfer 

5.  Conclusions  &  Challenges 
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I .  Issues 


Workforce 

Grey  Tsunami 

Continuity  of  knowledge  and  skill 

Army  TRADOC  “Continuity  Files" 

Moving  Target  Issues 

Continual  change  in  macrocognitive  work  and  its  technologic 

DoD  domains  in  particular 

Needs  for  adaptability,  flexibility,  transfer 
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I .  Issues 


“The  military  has  a  training  problem” 

(Chatham,  2009) 


•  Traditional  versus  “SSTR”  operations 

(Stability,  Security,  Transition,  Reconstruction,  DoD,  2005) 

•  “Today’s  missions.,, require  that  we  also  train  each  soldier 
to  be  a  little  bit  of  a  linguist,  anthropologist,  city  manager, 
arbitrator,  negotiator,  engineer,  contact  specialist, 
ambassador,  and  consummate  bureaucrat...  ’’  (Chatham,  2009) 
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I.  Issues 


* 


“New”  Directions  in  Expertise  Studies 


"The  chapters  (in  this  book)  make  it  clear  that  the  field  has 
built  upon,  yet  gone  beyond,  the  classic  research  studies  that 
compared  expert  and  novice  performance.  The  emphasis  has 
turned  to  the  development  of  expertise..."  (Bransford  & 
Sch\wartz,  2009). 


I .  Issues 


Accentuated  Needs 

•  Adaptability 

•  Flexibility 

•  Transfer 

•  Learning 

•  Continuity  -  Capability  Preservation  for  Sharing 
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2.  Empirical  Base:Academic  Studies 


Psychology  of  Learning 

Studies  of  long-term  retention 
Studies  of  transfer 

Proficiency  Scaling 

10,000  hours  is  misleading/misguided 
Initiate-apprentice-journeyman  levels,  in  particular 
Accelerate  across  the  apprentice-journeyman  gap 

Longitudinal  perspective 

Needs  for  adaptability,  flexibility,  transfer 


2.  Empirical  Base:Academic  Studies 


Deliberate  Practice 

Ericsson,  K.  A.,  Charness,  N.,  Feltovich,  R  J.,  &  Hoffman,  R.  R.  (2006). 
Cambridge  handbook  of  expertise  and  expert  performance.  Cambridge 
University  Press. 


Transfer  Appropriate  Processing 


Morris,  C.D.,  Bransford,  J.D.,  &  Franks,  J.J.  ( 1 977).  Levels  of  processing  versus 
transfer-appropriate  processing.  Journo/  ofVerbal  Learning  and  Verbal  Behavior, 
/  6,  5 1 9-533. 

Adaptive  Expertise 

Hatano,  G.and  K.  Inagaki  (1 986).  "Two  courses  of  expertise."  Child 
Development  and  Education  in  Japan,  pp.  262-272. 
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2.  Empirical  Base:  the  “Real  World” 


Training  Across  the  Proficiency  Scale 

Trainees  learn  more  from  successes  because  they  need  help 
figuring  out  what  to  do. 

Experienced  people  learn  more  from  failures  because  they 
have  already  mastered  the  routines  and  are  in  a  position  to 
make  use  of  the  information  found  in  errors. 

Hence,  the  value  of  Decision  Making  Exercises 


2.  Empirical  Base:  the  “Real  World” 


Successful  Cases  of  Acceleration 

•  Top  Gun 

•  National  Training  Center  (NTC) 

•  The  “first  training  revolution” 

•  “Engagement  Simulations”  (Chatham) 
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2.  Empirical  Base:  the  “Real  World” 


Field  Battle:  National  Training  Center  (“NTC”): 
Increased  Success  Rates,  Post  Training 


•  Light  infantry  platoons:  30: 1 

•  Combined  arms  teams:  15:1 

•  Regiments/Brigades  5:1 

(Duration:  3-4  Weeks  at  NTC) 
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Characteristics  of  Successful  Programs 


•  Independent,  representative,  and  superior  opposing 
(“red”)  force  (translates,  more  generally  to  “tough  cases”) 

•  Ability  to  measure  v^hat  happened 

•  As  high  fidelity  as  needed 

•  ARR.:  After  Action  Review: 

Objective,  detailed  feedback 

Situation  awareness,  decision  processes,  state  of  mind 
Participation  at  all  levels,  and  as  real  functional  units 
Self  Assessment  in  addition  to  external 

•Tailoring  to  the  learner  (individual  and/or  group), 
“customized” 
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3. Theories  &  Hypotheses 


Cognitive  Flexibility  Theory 

Spiro,  R.  J..  Feltovich.  P.  J.,  Jacobson,  M.  j.  and  Coulson.  R.  L  (1992).  Cognitive  flexibility, 
constructivism,  and  hypertext:  Random  access  instruction  for  advanced  knowledge  acquisition  in  ill- 
structured  domains.  In  T.  M.  Duffy  and  D.  H.  Johassen  (Es.),  Constructivism  and  the  technology  of 
instruction.A  conversation  (pp.  65-75).  Hilsdale,  NJ:  Lawrence  Eribaum  Associates. 


Cognitive  Transformation  Theory 

Klein,  G,  and  Baxter,  H.  C.  (2009).  Cognitive  Transformation  Theory;  Contrasting  Cognitive  and 
Behavioral  Learning.  In  The  PSI  Handbook  of  Virtual  Environments  for  Training  and  Education: 
Developments  for  the  Military  and  Beyond.  Vo/.  I:  Learning,  Requirements,  and  Metrics  (pp.  50-65).  Santa 
Barbara,  CA:  Praeger  Security  International. 


Common  Elements  Theory 

Thorndike,  E.  L  (1921).  Educational  psychology,  Volume  II:  The  psychology  of  learning.  New  York: 
Columbia  University  Teacher's  Coliege. 


3. Theories  &  Hypotheses 


Common  Elements  Theory 

Learning  and  transfer  have  to  do  with  the  strength  of  stimulus-response  bonds. 
These,  ultimately,  can  be  understood  as  being  neuronal  connections:  the  bonds 
are  "in  neurones"  (p.  354). Transfer  is  when  the  strengthening  of  one  S-R  bond 
also  strengthens  some  other  S-R  bond. 


Bond  A  (Original 
Task) 


Bond  B 
(Transfer  Task) 


Strengthened 

Weakened 

Strengethened 

"Positive  similar 
change,"  or 
facilitation 

Positive  opposite 
change,"  or 
inhibition 

Weakened 

"Negative 
opposite  change" 

"Negative  similar 
change" 
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3. Theories  &  Hypotheses 


Common  Elements  Theory 

"...a  change  in  one  function  alters  any  other  only  insofar  as  the  two 
functions  have  as  factors  identical  elements...  the  change  is  simply  the 
necessary  result  upon  the  second  function  of  the  alteration  of  those 
of  its  factors  which  were  elements  of  the  first  function,  and  so  were 
altered  by  its  training.  To  take  a  concrete  example,  improvement  in 
addition  will  alter  one's  ability  in  multiplication  because  addition  is 
absolutely  identical  with  a  part  of  multiplication,  and  because  certain 
other  processes,  e.g.,  eye  movements  and  the  inhibition  of  all  save 
arithmetical  impulses,  are  in  part  common  to  the  two  functions"  (pp. 
358-359). 


4.  Phenomena  of  Training  &  Transfer 


in  a  more  or  less  mysterious  way,  learning  to  do 
one  thing  well  will  make  one  do  better, 
things  that  in  concrete  appearance  have  absolutely 
no  community  with  it.” 

Thorndike,  1 92 1 ,  p.  364. 
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4.  Phenomena  of  Training  &  Transfer 


We  may  not  need  to  make  more  experts, 
but  we  do  need  to  study  them 
so  that  we  can  learn  from  them 


Deliberative  preparation 

Anticipation  of  difficulties  and  constraints  shapes  the  practice 
environment 

Constant  practice  at  performance  self-diagnosis. 


Eccles.  D.  W.  and  Ward,  P.  and  Woodman, T.  (2009).  Competition-specific  preparation 
and  expert  performance.  Psychology  of  Sport  and  Exercise.  1 0, 96- 1 07 


4.  Phenomena  of  Training  &  Transfer 


Paradoxical  Effects 

"So  a  man  trained  for  an  hour  on  a  typewriter  of  standard  keyboard 
might,  after  a  second  hour  of  practice,  with  the  keys  being 
changed...  do  worse  than  at  the  beginning:  but  if  he  practiced  an 
hour  daily  on  each  sort  of  machine  he  would  not  fall  back  to  his 
initial  score  for  long,  would  soon  come  to  be  able  to  turn  from  one 
to  the  other  system  of  bonds  at  the  mere  sight  of  the  machine,  and 
would  probably  find  that  20  hours  of  the  two  alternative  systems 
gave  greater  ability  at  either  than  ten  hours  of  practice  at  each 
alone  would  have  given"  (pp.  356-357). 
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4.  Phenomena  of  Training  &  Transfer 


Paradoxical  Effects 


•  The  easy,  fast  way  may  not  get  you  there 

•  Short  results  versus  long  term  retention  and  transfer 

•  Some  things  that  make  learning  harder  also  make  it  better 

•  Bjork's  "desirable  difficulties"  in  learning 

Paradoxical  Situations 

•The  paradox  of  Deliberate  Practice  -  the  more  limited  the  focus,  the 
better  the  training  but  the  less  you  can  cover. 


4.  Phenomena  of  Training  &  Transfer 


Mistakes  and  Near  Misses  Can  be  Good 


•  Deliberate  practice--  always  “stretching” 

•  Olympic  skaters  fall  down  in  practice 

•After Action  Reviews, 

•  Morbidity  and  Mortality  Conferences, 

•  “Think-Like-a-Commander,” 

•  High  Reliability  Organizations, 

•  Tough  Cases... 
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5.  Conclusions  and  Challenges 


When  the  notion  of  “task”  no  longer  applies. . . . 

Goals  morph 
Tools  morph 
Emerging  surprises 
Constant  multitasking 


Cyberwar 
UxV  operation 


5.  Conclusions  and  Challenges 


Assumptions  about  the  concept  of  “task”  in  ALL  theories  and 
hypotheses  about  transfer: 

Isolable 

Fixed,  stable  "elements” 

Stable  goals 
Proceduralizable 

Moving  Target  Problems 

Some  jobs,  such  as  piloting,  involve  constant  and  intense  multitasking. 
Focusing  serially  on  the  achievement  of  goals  can  even  be  dangerous. 
The  worker  must  do  parallel  attention. 

Retention  and  transfer  training,  by  tradition,  have  focused  on  the 
idea  of  isolated,  individual  skills,  rather  than  skill  at  multitasking  per  se. 
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5.  Conclusions  and  Challenges 


Measurement  Issues 


HEAT  measurement  (hits,  errors,  accuracy,  time) 
works  form  some  jobs/tasks,  but  not  for  many  others 
System-level  measurement. 

Moving  Target  Problems 

The  problem  of  “representative  tasks” 

The  problem(s)  of  control. 

Measurement  at  higher-levels  of  proficiency  (journeyman,  expert) 
where  performance  is  near  ceiling 


5.  Conclusions 


Current  propositions  concerning  training 


INITIATE 

ExpenentKif  teeming 

The  trainee  first  experiences  learning  content  through  hands-on  practice  and  then 
reflects  upon  that  experience  while  trying  to  abstract  key  ideas. 

Can  rely  on  simulation  systems. 

Learning  from  Errors 

Modifying  the  sccnanos  within  the  simulation  such  chat  they  emphasize  common 
mrsconceptions.  that  is.  correct  reductive  or  flawed  mental  models. 

Probtem-boied  Learning 

Scenarios  are  populated  with  more  challenging  content. 


APPRENTICE 

Situated  Leamng  and  Cognitive  Ffexibi/rty 

Actions  and  decisions  are  practiced  in  the  context  of  the  operational  environment, 
and  when  warranted,  the  learner  is  guided  in  his/her  own  performance.  This  could 
be  implemented  with  simulation-based  environments  or  practiced  in  the  field. 
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5.  Conclusions 


Current  propositions  concerning  training 


JOURNEYMAN 

Cognitive  Transformation  and  Cognitme  Flexibitity 

The  (earner  experiences  highly  variable  tasks  and  rare,  difficult  examples  that 
challenge  the  learner's  concepts  and  categories. 

Given  their  increasing  autonomy,  those  at  the  journeymen  level  might 
independently  use  simulation«based  environments  to  tram  on  these  types  of 
teaming  situations. 

Note,  though,  that  the  learner  must  recognixe  incorrect  understanding  or 
assumptions  and  resolve  any  misconceptions  he  or  she  may  hold.  Mentoring  is 
critical  at  this  point 


5.  Conclusions 


OYou  need  KE  for  continuity 
KE  Case  Corpus 

Tough  Case  corpus  enables  "time  compression" 


OYou  need  to  cope  with  the  Moving  Target 
Invert  the  Common  Elements  Theory 
Hiatus  (maintenance)  training  focuses  on  what  is 
different  or  changing 
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Quick  relevant  observations  with 
situated  assessment 

R.  Hubal 

RTI  International 
09-10  October  2009 


Background  for  one  study 


■  Many  inner-city  adolescents  are  at  risk  during  social  situations 
that  require: 

♦  Emotional  control. 

♦  Conflict  resolution  through  negotiation,  seeking 
information,  expressing  preferences. 

♦  Adverse  consequences  include  violence,  school  suspension, 
criminal  activity. 

Training  on  these  skills  is  well  studied,  but... 

■  Assessment  is  unsatisfactory: 

♦  Want  to  assess  adolescent's  learning  before  he's  "on  the 
street". 

♦  Existing  methods  use  surveys  or  text-based  cases,  not 
"assessment-by-doing". 
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■  Vignette  storylines  designed  to  target  skills: 

♦  That  is,  competencies  -  engaging  in  negotiation, 
maintaining  emotional  control  and  being  non¬ 
provocative,  seeking  information. 

■  Language  models  designed  to  interpret  participant  input  as 
reflecting  one  or  more  social  skill. 

■  Behavior  models  designed  so  that  the  virtual  character 
attempts  different  strategies  to  require  participants  to 
demonstrate  skills. 

■  Methods  devised  to  measure  participant  engagement. 


■  Procedure; 

♦  Participant  given  pre-test  set  of  vignettes. 

Wizard-of-Oz  methodology. 

Monitor  how  participants  handle  vignettes  designed  to  elicit 
risky  behavior. 

♦  Participant  given  standard  text-based  assessments. 

♦  Participant  shown  PACT  video  in  presence  of  facilitator. 

♦  After  up  to  6  month  delay,  participant  given  post-test  set  of 
vignettes  (again  WoZ). 

♦  Behavioral  reports,  teacher  ratings,  self-reports  collected. 
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Methods,  continued 


■  Measures: 

♦  Engagement  with  vignettes. 

♦  Body  language. 

♦  Verbalizations. 

♦  Number  of  conversational  turns. 

♦  Response  time. 

♦  Outcome: 

Positive  outcome  is  to  decline  or  back  away. 

Negative  outcome  is  to  agree  or  escalate 
confrontation. 


■  Scoring  procedures  used  during 
pre/post  test  with  vignettes 
identify: 

♦  Level  of  emotional  control. 

♦  Interpersonal 
communication  skills. 

♦  Analysis  against 
established  measures 
provides  some  support  for 
construct  and  criterion 
validity. 

■  Nearly  all  queried  participants 
stated  their  virtual  decision 
mirrored  what  would  be  their 
real-life  decision. 

Note:  acceptability  /  usability 
not  different  among  groups. 


■  Participants  agree  to  help/go 
along  with/escalate 
confrontation  with  the  synthetic 
character  one-third  to  half  of 
time. 

■  Simulation  effective  in 
differentiating: 

Adolescents  with  clinical 
Conduct  Disorder. 

•  Adolescents  who  had 
participated  in  live  training 
sessions  on  key  skills. 
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■  Design  of  vignettes: 

>  Set  of  vignettes  needs  to  encompass  competencies. 

Thus,  need  to  have  theory  defining  competencies. 

♦  Need  to  be  realistic  (engaging). 

"Representative  tasks": 

♦  Need  to  be  realistic  (relevant). 

♦  Consider  context  -  language,  clothing,  gestures. 

The  experience  in  location  A  should  equate  to  the  experience 
in  location  B. 

♦  Describe  research  as  about  competent  performance,  not 
"expertise". 

■  More  in  one-page  concept  paper ... 
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ELECTRIC  POWER 
RESEARCH  INSTITUTE 


Electric  Utility  Super-Experts 

Acceleration  and  Retention 
issues 


David  Ziebell 
dziebell@epri.com 


Goals  for  this  presentation... 


A  little  about  EPRI  and 
EPRI’s  work  on  a  particular 
subset  of  knowledge 
management;  Capturing 
tacit  knowledge 

-  Evolution  toward  expert 
knowledge  and 

Industry  examples 


A  speculation 
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Our  History... 


Founded  by  and  for  the  electricity 
industry  in  1973 


Independent,  nonprofit  center  for 
public  interest  energy  and 
environmental  research 


Collaborative  resource  for  the 
electricity  sector 

Major  offices  in  Palo  Alto,  CA; 
Charlotte,  NC;  Knoxville,  TN 

-  Laboratories  in  Knoxville, 
Charlotte  and  Lenox,  MA 


Chauncey  Starr 
EPRI  Founder 
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Our  Members... 


•  450+  participants  in  more  than  40 
countries 

•  EPRI  members  generate  more 
than  90%  of  the  electricity  in  the 
United  States 

•  International  funding  of  more  than 
15%  of  EPRI’s  research, 
development  and  demonstrations 

•  Programs  funded  by  more  than 
1,000  energy  organizations 
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Our  Role... 


Help  Move  Technologies  to  the  Commercialization  Stage. 


A 


Basic 
Research 
and 
Development 


National 

Laboratories 

Universities 


EPRI 


Suppliers 

Vendors 


Technology  Accelerator! 
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Portfolio  Spans  the  Entire  Electricity  Sector 


Generation 

•  Advanced  Coal  Plants. 
Carbon  Capture  and  Storage 

•  Combustion  Turbines 

•  Environmental  Controls 

•  Generation  Planning 

•  Major  Component  Reliability 

•  Operafions  and  Maintenance 

•  Renewables 


Nuclear  Power 

■  Advanced  Nuclear 
Technology 

■  Chemistry.  Low-Level  Waste 
and  Radiation  Management 

■  Equipment  Reliability 

■  Fuel  Reliability 

•  Instrumentation  and  Control 

•  Matenal  Degradation/Aging 

•  Nondestructive  Evaluation 
and  Material  Characterization 

■  Risk  and  Safety 
Management 

■  Used  Fuel  and  High-Level 
Waste  Mahagement 


Power  Delivery  &  Utilization 

•  Transmission  Lines  and 
Substations 

•  Grid  Operations  and  Planning 

•  Distrlbubon 

■  Energy  Utilization 

•  Cross  Cutting  Technologies 


Environment 

•  Air  Quality 

•  Global  Climate  Change 

•  Land  and  Groundwater 

•  Occupational  Health  and 
Safety 

•  T&D  Environmental  Issues 
■  Water  and  Ecosystems 
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[jj  EPRI  Research  Results  and  Methods 


2000 

1004658 

Capturing  Undocumented  Worker-Job-Knowledge; 
Overview  and  2000  Status  Report 

2002 

1004663 

Guidelines  for  capturing  valuable  undocumented 
knowledge  from  energy  industry  personnel 

2002 

1002896 

Capturing  and  using  high-value  undocumented  knowledge 
in  the  nuclear  industry — guidelines  and  methods. 

2004 

1009581 

Real-Time  Expert  Knowledge  Acquisition  &  Transfer 

2008 

1016710 

Accelerating  the  Achievement  of  Mission-Critical 
Expertise 

■ 

U  Actual  challenges  met  by  EPRI’s  method... 


•  Transmission  System  Protection  Expert  is  retiring  and 
he’s  the  only  person  who  can  quickly  and  reliably 
diagnose  weird  protection  system  operations 

•  Fractionated  semi-computerized  billing  system  still  in 
service  but  the  guru  is  moving  on 

•  Chief  System  Operator  who  has  held  the  ECC  together 
during  several  corporate  transitions,  including 
incorporating  1995  OSHA  requirements  (for  violations  of 
same,  industry  people  have  been  jailed)  has  retired 

•  Young  underground  station  operators  just  don’t  seem  to 
“get  it”  about  how  to  respond  to  common  events,  and  the 
operating  engineer  (smart,  two  years  with  company)  has 
taken  over  for  the  promoted/transferred  prior  expert 
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More  examples  ... 


•  Senior  VP  special  projects  -  called  for  disasters  large  and 
small 

•  Chief  operator  of  natural  gas  liquification  plant  nearing 
retirement  just  as  system  re-design  is  needed 

•  Engineering  Manager  facing  high  turnover  in  an  aging 
system  facing  a  growth  spurt  -  long  range  planning 
process  needs 
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^  Insight:  A  possible  gap  in  traditional  methods 


Three  Kinds  of  Expertise  Needed 


Leadership 


Succession  Planning 


Key  Technical  Wisdom 


Productive  Workforce 


Wait  while 
experience  grows? 

Workforce  Planning, 
Training,  Information 
Technology, 
Performance  Tools 


Traditionai  methods  address  two 
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EPRI’s  tacit  knowledge  method  works  2  ways 


•  Middle-of-the-bell-curve  Undocumented  Knowledge 
Capture 

-  Often  can  be  captured  efficiently  and  rather  easily 

-  Can  (should)  be  made  widely  available  via  training  or 
digital  tools:  Transfer  \s  well  understood,  systematic 


•  High-end  Technical  Wisdom  Capture 

-  Often  requires  much  effort  to  capture 

-  Successful  transfer  to  a  few  specified  individuals  is 
proven,  but  not  yet  systematic 

-  The  value  is  at  the  fringes  of  the  discipline  -  where 
resilience  lives 
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P  Example  “Big  Picture”  for  mass  consum 
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Example  -  LNG  Operations 
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Example  “Expert  Thought  Process”  Transfer 


*  Captured  tacit  knowledge 
organized 

•  Process  is  a  basis  for 
expert  judgment 

-  Not  simply  facts,  but 
transformed  knowledge 

-  Demonstrates  the 
adaptivity  we  seek  in 
our  experts 


evpenence  situation  in  a 
dianging  environment 

tt  assessed  oy 
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Where  are  the  “Agents”  and  what  can  they  do? 


Permh^ion 


The  Intelligent-Acting  Enterpnse 

Inteihgent  Acting 
Personnel 

PfnoDBfl  Deal  Direeth 
wifb  Onidde  World 


Structural  Knowledge  Assets 


Rrtnll^ 

riooi 

InlelligfDi 

Arliog 

Prrsoniifl 

and 

fiom 

utieiligeui 

.\pplicatioo 

of 

Slnirniral 

Knonirdgr 

AmMs 


Diiort 
Salft 
of 

Sducraral 

Knonlfdgr 

Aston 


From  ‘Knowledge  Managameni  An  Emerging  C>«opir>e  Rooted  m  a  Long  laatoY  Kan  M  VV«g 
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A  Speculation 


•  Agents  are  of  two  kinds 

-  the  knowledge  worker,  middle  of  the  bell  curve 
producer  that  does  repetitive  pre-planned  work  in  a 
quasi-stable  system.  Some  knowledge  workers  are 
experts  who  are  allowed  to  improve  the  efficiency  of 
the  pre-planned  work  within  the  quasi-stable  system 

-  The  super  expert,  having  agency/authority  (granted  or 
tacit)  to  change  the  quasi-stable  system  and  it’s 
relation  to  the  outside  world. 

•  Institutions  are  dependent  on  these  super-experts 
for  resiliency  but  don’t  often  explicitly  act  like  this. 


C  200a  Ciactx  >nc  M 


^6 


(=f=iai|::r.f.’^‘- 


Potential  Barriers  in  Utility  Space 


Non-standard  problems  -  chaotic  or  merely  complex? 
Difficult  problems  rare  -  25-35  years  to  see  enough? 
Complexity  plus  longevity  (of  the  technology) 

Biggest  problems  occur  at  the  fringes  of  a  discipline  -  the 
fog  of  society 


Complex 

P-S-R 


Emergent 

Pracnce 

Ctiaob'c 

A-S-R 

Novel 

Practice 


Complicated 
S-A-R 
Good 

le^^^Practice 

tfeVsiri 


Simple 

S-C-R 


Sest 

Practice 
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